hadoop - Spark Streaming with large messages java.lang.OutOfMemoryError: Java heap space -


enter image description herei using spark streaming 1.6.1 kafka0.9.0.1 (createstreams api) hdp 2.4.2, use case sends large messages kafka topics ranges 5mb 30 mb in such cases spark streaming fails complete job , crashes below exception.i doing dataframe operation , saving on hdfs in csv format, below code snippet

reading kafka topic:      val lines = kafkautils.createstream[string, string, stringdecoder, stringdecoder](ssc, kafkaparams, topicmap, storagelevel.memory_and_disk_ser_2/*memory_only_ser_2*/).map(_._2)   writing on hdfs:       val hdfsdf: dataframe = getdf(sqlcontext, eventdf, schema,topicname)       hdfsdf.show       hdfsdf.write         .format("com.databricks.spark.csv")         .option("header", "false")         .save(hdfspath + "/" + "out_" + system.currenttimemillis().tostring())  16/11/11 12:12:35 warn receivertracker: error reported receiver stream 0: error handling message; exiting - java.lang.outofmemoryerror: java heap space     @ java.lang.stringcoding$stringdecoder.decode(stringcoding.java:149)     @ java.lang.stringcoding.decode(stringcoding.java:193)     @ java.lang.string.<init>(string.java:426)     @ java.lang.string.<init>(string.java:491)     @ kafka.serializer.stringdecoder.frombytes(decoder.scala:50)     @ kafka.serializer.stringdecoder.frombytes(decoder.scala:42)     @ kafka.message.messageandmetadata.message(messageandmetadata.scala:32)     @ org.apache.spark.streaming.kafka.kafkareceiver$messagehandler.run(kafkainputdstream.scala:137)     @ java.util.concurrent.executors$runnableadapter.call(executors.java:511)     @ java.util.concurrent.futuretask.run(futuretask.java:266)     @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1142)     @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:617)     @ java.lang.thread.run(thread.java:745)

followed :

 java.lang.exception: not compute split, block input-0-1478610837000 not found @ org.apache.spark.rdd.blockrdd.compute(blockrdd.scala:51) @ org.apache.spark.rdd.rdd.computeorreadcheckpoint(rdd.scala:313) @ org.apache.spark.rdd.rdd.iterator(rdd.scala:277) @ org.apache.spark.rdd.mappartitionsrdd.compute(mappartitionsrdd.scala:38) @ org.apache.spark.rdd.rdd.computeorreadcheckpoint(rdd.scala:313) @ org.apache.spark.rdd.rdd.iterator(rdd.scala:277) @ org.apache.spark.scheduler.resulttask.runtask(resulttask.scala:66) @ org.apache.spark.scheduler.task.run(task.scala:89) @ org.apache.spark.executor.executor$taskrunner.run(executor.scala:214) @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1142) @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:617) @ java.lang.thread.run(thread.java:745) driver stacktrace:


Comments

Post a Comment

Popular posts from this blog

c# SetCompatibleTextRenderingDefault must be called before the first -

i tried search exception couldn't find solution on case i'am using code below invoke .net application : assembly assem = assembly.load(data); methodinfo method = assem.entrypoint; var o = activator.createinstance(method.declaringtype); method.invoke(o, null); the application invoked has form , in entrypoint of application : [stathread] static void main() { application.enablevisualstyles(); application.setcompatibletextrenderingdefault(false); //exception application.run(new form1()); } setcompatibletextrenderingdefault must called before first iwin32window object created in application. edit : assembly = assembly.load(data); methodinfo method = a.gettype().getmethod("start"); var o = activator.createinstance(method.declaringtype); method.invoke(o, null); you should create new method skips initialization...
Read more

C#.NET Oracle.ManagedDataAccess ConfigSchema.xsd -

i'm not able find config schema file (configschema.xsd) of oracle.manageddataaccess itellisense in application configuration file. it supposed in odp.net managed driver files cannot find it... configschema.xsd xml schema file defines configuration section odp.net, managed driver. https://docs.oracle.com/cd/e48297_01/doc/win.121/e41125/installodpmd.htm after having installed oracle developer tools visual studio 2015, seems 2 schema files have been linked app.config file : oracle.dataaccess.common.configuration.section.xsd oracle.manageddataaccess.client.configuration.section.xsd both stored in : c:\program files (x86)\microsoft visual studio 14.0\xml\schemas
Read more

c++ - Fill runtime data at compile time with templates -

i have specific situation in prepare runtime structures @ compile time without need duplicate code. i have 2 structs use register @ compile time types compiler wrote: using typeid = u8; template<typename t, typename type_id, type_id i> struct typehelper { static constexpr type_id value = std::integral_constant<type_id, i>::value; }; template<typename t> struct type : typehelper<t, u8, __counter__> { static_assert(!std::is_same<t,t>::value, "must specialize type!"); }; these used in config header macro specialize type<t> multiple types require: using type_size = unsigned char; #define get_nth_macro(_1,_2,_3, name,...) name #define register_type(...) get_nth_macro(__va_args__, register_type3, register_type2, register_type1)(__va_args__) #define register_type1(_type_) register_type2(_type_, _type_) #define register_type2(_type_,_name_) \ constexpr typeid type_##_name_ = __counter__; \ template<> struct type<_type_...
Read more