python - Kafka consumer: AttributeError: 'list' object has no attribute 'map' -


i want read messages kafka queue in python. example, in scala it's quite easy do:

    val ssc = new streamingcontext(conf, seconds(20))      // divide topic partitions     val topicmessages = "mykafkatopic"     val topicmessagesmap = topicmessages.split(",").map((_, kafkanumthreads)).tomap      val messages = kafkautils.createstream(ssc, zkquorum, group, topicmessagesmap).map(_._2)      messages.foreachrdd { rdd =>       //...     }

i want same in python. current python code:

from pyspark.streaming import streamingcontext pyspark.streaming.kafka import kafkautils  ssc = streamingcontext(sc, 20)  topicmessages = "mykafkatopic" topicmessagesmap = topicmessages.split(",").map((_, kafkanumthreads)).tomap  messages = kafkautils.createstream(ssc, zkquorum, group, topicmessagesmap)

however error @ line topicmessagesmap = topicmessages.split(",").map((_, kafkanumthreads)).tomap:

attributeerror: 'list' object has no attribute 'map'

how make code working?

update:

if run code in jupyter notebook, error shown below:

messages = kafkautils.createstream(ssc, zkquorum, "spark-streaming-consumer", {inputkafkatopic: list})

spark streaming's kafka libraries not found in class path. try 1 of following.

  1. include kafka library , dependencies in spark-submit command as

    $ bin/spark-submit --packages org.apache.spark:spark-streaming-kafka-0-8:2.0.0 ...

  2. download jar of artifact maven central http://search.maven.org/, group id = org.apache.spark, artifact id = spark-streaming-kafka-0-8-assembly, version = 2.0.0. then, include jar in spark-submit command as

    $ bin/spark-submit --jars ...

do understand correctly way make working use spark-submit , it's impossible run code jupyter/ipython?


Comments

Post a Comment

Popular posts from this blog

php - How to add and update images or image url in Volusion using Volusion API -

actually have store importing products store product data insert , update image or image url not insert , update. this xml code name datapro.txt. <products_joined> <productcode>3710_012t</productcode> <vendor_partno>eah5450silent/di/1gd3(lp)</vendor_partno> <productname>test product ta</productname> <hideproduct>n</hideproduct> <stockstatus>20</stockstatus> <lastmodified>1/5/2016 10:25:00 am</lastmodified> <lastmodby>2</lastmodby> <productweight>0.9</productweight> <productprice>100</productprice> <productmanufacturer>asus tek</productmanufacturer> <vendor_price>32.69</vendor_price> <numproductssharingstock>0</numproductssharingstock> <categoryids>107</categoryids> <producturl>http://tebkq.mvlce.servertrust.com/productdetails.asp?productcode=3710_012t</producturl...
Read more

Laravel mail error `Swift_TransportException in StreamBuffer.php line 269: Connection could not be established with host smtp.gmail.com [ #0]` -

i developing website contact-us page guest users can send me email using gmail . anytime try send mail using contact-us form, error swift_transportexception in streambuffer.php line 269: connection not established host smtp.gmail.com [ #0] this mail.php return [ 'driver' => env('mail_driver', 'smtp'), 'host' => env('mail_host', 'smtp.gmail.com'), 'port' => 587, 'from' => ['address' => '', 'name' => ''], 'encryption' => env('mail_encryption', 'tls'), 'username' => env('mail_username'), 'password' => env(mail_password), 'sendmail' => '/usr/sbin/sendmail -bs', 'pretend' => env('mail_pretend', false), ]; this .env mail_driver=smtp mail_host=smtp.gmail.com mail_port=587 mail_username=afecjaped@gmail.com mail_password=********...
Read more

c# SetCompatibleTextRenderingDefault must be called before the first -

i tried search exception couldn't find solution on case i'am using code below invoke .net application : assembly assem = assembly.load(data); methodinfo method = assem.entrypoint; var o = activator.createinstance(method.declaringtype); method.invoke(o, null); the application invoked has form , in entrypoint of application : [stathread] static void main() { application.enablevisualstyles(); application.setcompatibletextrenderingdefault(false); //exception application.run(new form1()); } setcompatibletextrenderingdefault must called before first iwin32window object created in application. edit : assembly = assembly.load(data); methodinfo method = a.gettype().getmethod("start"); var o = activator.createinstance(method.declaringtype); method.invoke(o, null); you should create new method skips initialization...
Read more