python - Scrapy: Error 1241 and 1064 while sending datas to MySQL -


i want send datas i've scrapped mysql database. here pipelines.py :

class myimagespipeline(imagespipeline):      def get_media_requests(self, item, info):         image_url in item['image_urls']:             yield scrapy.request(image_url)      def item_completed(self, results, item, info):         image_paths = [x['path'] ok, x in results if ok]         item['image_paths'] = image_paths         return item   class mysqlpipeline(object):      def __init__(self):         self.conn = mysqldb.connect(user='testuser', passwd='megablock:333', db='crawl', host='localhost', charset="utf8", use_unicode=true)         self.cursor = self.conn.cursor()       def process_item(self, item, spider):              guid = self._get_guid(item)         = datetime.utcnow().replace(microsecond=0).isoformat(' ')          try:             self.cursor.execute("""insert produits (guid, now, product, link, price, description, image_urls, image_paths, brand, couleur, gamme, largeur, profondeur, hauteur, longueur, diametre)                values (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)""",              (guid, now, item['product'], item['link'], item['price'], item['description'], item['image_urls'], item['image_paths'], item['brand'], item['couleur'], item['gamme'], item['largeur'], item['profondeur'], item['hauteur'], item['longueur'], item['diametre']))              self.conn.commit()             print "data sent db"          except mysqldb.error, e:             print "error %d: %s" % (e.args[0], e.args[1])           return item      def _get_guid(self, item):         """generates unique identifier given item."""         # hash based solely in url field         return md5(item['link']).hexdigest()

here item.py :

class ikeacrawlitem(scrapy.item):     product = scrapy.field()     link = scrapy.field()     price = scrapy.field()     description = scrapy.field()     image_urls = scrapy.field()     images = scrapy.field()     brand = scrapy.field()     couleur = scrapy.field()     gamme = scrapy.field()     largeur = scrapy.field()     profondeur = scrapy.field()     hauteur = scrapy.field()     longueur = scrapy.field()     diametre = scrapy.field()     image_paths = scrapy.field()

and, if needed, database structure :

create table produits (   guid char(32) primary key,   product varchar(30),   link text,   price float(5,2) unsigned,   description varchar(100),   image_urls text,   image_paths text,   brand varchar(30),   couleur varchar(150),   gamme text,   largeur float(5,2) unsigned,   profondeur float(5,2) unsigned,   hauteur float(5,2) unsigned,   longueur float(5,2) unsigned,   diametre float(5,2) unsigned,   datetime ) default charset=utf8;

the scrapping works well. i've send 3 first elements alone, works too, suggesting it's not problem of mysql connection. thought might linked concurrency issue, slow scrapping, doesn't work. still, here error message receive :

2016-11-08 11:16:41 [scrapy] debug: crawled (200) <get http://www.ikea.com/fr/fr/catalog/products/30262095/> (referer: http://www.ikea.com/fr/fr/catalog/productsaz/6/) 2016-11-08 11:16:41 [scrapy] debug: file (uptodate): downloaded image <get http://www.ikea.com/fr/fr/images/products/gunnern-gueridon-blanc__0242398_pe381794_s4.jpg> referred in <none> error 1241: operand should contain 1 column(s) 2016-11-08 11:16:41 [scrapy] debug: scraped <200 http://www.ikea.com/fr/fr/catalog/products/30262095/> {'brand': 'ikea',  'couleur': [u'blanc', u'gris'],  'description': u'gu\xe9ridon',  'diametre': 0.0,  'gamme': u'\xc9tag\xe8res',  'hauteur': u'74.7',  'image_paths': 'images',  'image_urls': [u'http://www.ikea.com/fr/fr/images/products/gunnern-gueridon-blanc__0242398_pe381794_s4.jpg'],  'images': [{'checksum': '70fce09525b1489155a34cbc55ce6729',              'path': 'full/275b007d3103babddff8e46fdde102d42fcae8a1.jpg',              'url': 'http://www.ikea.com/fr/fr/images/products/gunnern-gueridon-blanc__0242398_pe381794_s4.jpg'}],  'largeur': 0.0,  'link': 'http://www.ikea.com/fr/fr/catalog/products/30262095/',  'longueur': 0.0,  'price': u'35',  'product': u'gunnern',  'profondeur': 0.0} 2016-11-08 11:16:45 [scrapy] debug: crawled (200) <get http://www.ikea.com/fr/fr/catalog/products/10288195/> (referer: http://www.ikea.com/fr/fr/catalog/productsaz/6/) 2016-11-08 11:16:45 [scrapy] debug: file (uptodate): downloaded image <get http://www.ikea.com/fr/fr/images/products/gunnern-armoire-verrouillable-rouge__0275675_pe413909_s4.jpg> referred in <none> error 1064: have error in sql syntax; check manual corresponds mysql server version right syntax use near '), 'meubles à miroir', '32', '10.4', '32', 0, 0)' @ line 2 2016-11-08 11:16:45 [scrapy] debug: scraped <200 http://www.ikea.com/fr/fr/catalog/products/10288195/> {'brand': 'ikea',  'couleur': [],  'description': u'armoire verrouillable',  'diametre': 0.0,  'gamme': u'meubles \xe0 miroir',  'hauteur': u'32',  'image_paths': 'images',  'image_urls': [u'http://www.ikea.com/fr/fr/images/products/gunnern-armoire-verrouillable-rouge__0275675_pe413909_s4.jpg'],  'images': [{'checksum': 'c95f7095c0b6645b5780bff39b47e3d2',              'path': 'full/20c2afef8324d798baac384970e9efb4557f9f19.jpg',              'url': 'http://www.ikea.com/fr/fr/images/products/gunnern-armoire-verrouillable-rouge__0275675_pe413909_s4.jpg'}],  'largeur': u'32',  'link': 'http://www.ikea.com/fr/fr/catalog/products/10288195/',  'longueur': 0.0,  'price': u'25',  'product': u'gunnern',  'profondeur': u'10.4'}

i have 2 different error message, don't understand why. along whole scrapping process, appeared alternatively. can please me ?

thank you


Comments

Post a Comment

Popular posts from this blog

php - How to add and update images or image url in Volusion using Volusion API -

actually have store importing products store product data insert , update image or image url not insert , update. this xml code name datapro.txt. <products_joined> <productcode>3710_012t</productcode> <vendor_partno>eah5450silent/di/1gd3(lp)</vendor_partno> <productname>test product ta</productname> <hideproduct>n</hideproduct> <stockstatus>20</stockstatus> <lastmodified>1/5/2016 10:25:00 am</lastmodified> <lastmodby>2</lastmodby> <productweight>0.9</productweight> <productprice>100</productprice> <productmanufacturer>asus tek</productmanufacturer> <vendor_price>32.69</vendor_price> <numproductssharingstock>0</numproductssharingstock> <categoryids>107</categoryids> <producturl>http://tebkq.mvlce.servertrust.com/productdetails.asp?productcode=3710_012t</producturl>
Read more

javascript - jQuery UI Splitter/Resizable for unlimited amount of columns -

using jqueryui splitter or resizable functions want create layout allows unlimited amount of columns added , columns need resizable. so fiddle: https://jsfiddle.net/f3glh3mw/ but in fiddle html contains out of nested sets instead of inline. limits possibilities above fiddle can have equal widths 2,4,8,16,32 etc columns. in between shows crooked can see in fiddle. when researching came on question: jquery ui , splitter if @ top answer shows structure like: <div class="wrap"> <div class="resizable resizable1"></div> <div class="resizable resizable2"></div> </div> full fiddle: http://jsfiddle.net/8qztj/86/ which structure want javascript hardcoded 2 resizable items. so here question: how modify last fiddle works amount of .resizable divs. or know of plugin can this. thanks helping! so far found plugin: http://www.bacubacu.com/colresizable/ which says , close functionality want. in
Read more

javascript - IE9 error '$'is not defined -

Image
<!--[if lte ie 9]><html class="lte9"><![endif]--><!--[if gt ie 9]><html><![endif]--> <!--[if !ie]><!--> <html> <!--<![endif]--> blah blah <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.12.1/jquery.min.js"></script> <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"></script> <script src="{{ asset('js/mypage.js') }}" type="text/javascript"></script> </body> </html> when load page, ie9 brings error '$' not defined . how can solve error? +++ mypage.js $(function(){ //when load page, //ie9 tag not moving. $(window).load(function() { $('div.mask').hide(); }); // when click , willshow show. //for not ie 9 if (!$('html').hasclass('lte9')) { var = $('ul.about > li > span.activebtn');
Read more