python - Scrapy: Error 1241 and 1064 while sending datas to MySQL -
i want send datas i've scrapped mysql database. here pipelines.py
:
class myimagespipeline(imagespipeline): def get_media_requests(self, item, info): image_url in item['image_urls']: yield scrapy.request(image_url) def item_completed(self, results, item, info): image_paths = [x['path'] ok, x in results if ok] item['image_paths'] = image_paths return item class mysqlpipeline(object): def __init__(self): self.conn = mysqldb.connect(user='testuser', passwd='megablock:333', db='crawl', host='localhost', charset="utf8", use_unicode=true) self.cursor = self.conn.cursor() def process_item(self, item, spider): guid = self._get_guid(item) = datetime.utcnow().replace(microsecond=0).isoformat(' ') try: self.cursor.execute("""insert produits (guid, now, product, link, price, description, image_urls, image_paths, brand, couleur, gamme, largeur, profondeur, hauteur, longueur, diametre) values (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)""", (guid, now, item['product'], item['link'], item['price'], item['description'], item['image_urls'], item['image_paths'], item['brand'], item['couleur'], item['gamme'], item['largeur'], item['profondeur'], item['hauteur'], item['longueur'], item['diametre'])) self.conn.commit() print "data sent db" except mysqldb.error, e: print "error %d: %s" % (e.args[0], e.args[1]) return item def _get_guid(self, item): """generates unique identifier given item.""" # hash based solely in url field return md5(item['link']).hexdigest()
here item.py
:
class ikeacrawlitem(scrapy.item): product = scrapy.field() link = scrapy.field() price = scrapy.field() description = scrapy.field() image_urls = scrapy.field() images = scrapy.field() brand = scrapy.field() couleur = scrapy.field() gamme = scrapy.field() largeur = scrapy.field() profondeur = scrapy.field() hauteur = scrapy.field() longueur = scrapy.field() diametre = scrapy.field() image_paths = scrapy.field()
and, if needed, database structure :
create table produits ( guid char(32) primary key, product varchar(30), link text, price float(5,2) unsigned, description varchar(100), image_urls text, image_paths text, brand varchar(30), couleur varchar(150), gamme text, largeur float(5,2) unsigned, profondeur float(5,2) unsigned, hauteur float(5,2) unsigned, longueur float(5,2) unsigned, diametre float(5,2) unsigned, datetime ) default charset=utf8;
the scrapping works well. i've send 3 first elements alone, works too, suggesting it's not problem of mysql connection. thought might linked concurrency issue, slow scrapping, doesn't work. still, here error message receive :
2016-11-08 11:16:41 [scrapy] debug: crawled (200) <get http://www.ikea.com/fr/fr/catalog/products/30262095/> (referer: http://www.ikea.com/fr/fr/catalog/productsaz/6/) 2016-11-08 11:16:41 [scrapy] debug: file (uptodate): downloaded image <get http://www.ikea.com/fr/fr/images/products/gunnern-gueridon-blanc__0242398_pe381794_s4.jpg> referred in <none> error 1241: operand should contain 1 column(s) 2016-11-08 11:16:41 [scrapy] debug: scraped <200 http://www.ikea.com/fr/fr/catalog/products/30262095/> {'brand': 'ikea', 'couleur': [u'blanc', u'gris'], 'description': u'gu\xe9ridon', 'diametre': 0.0, 'gamme': u'\xc9tag\xe8res', 'hauteur': u'74.7', 'image_paths': 'images', 'image_urls': [u'http://www.ikea.com/fr/fr/images/products/gunnern-gueridon-blanc__0242398_pe381794_s4.jpg'], 'images': [{'checksum': '70fce09525b1489155a34cbc55ce6729', 'path': 'full/275b007d3103babddff8e46fdde102d42fcae8a1.jpg', 'url': 'http://www.ikea.com/fr/fr/images/products/gunnern-gueridon-blanc__0242398_pe381794_s4.jpg'}], 'largeur': 0.0, 'link': 'http://www.ikea.com/fr/fr/catalog/products/30262095/', 'longueur': 0.0, 'price': u'35', 'product': u'gunnern', 'profondeur': 0.0} 2016-11-08 11:16:45 [scrapy] debug: crawled (200) <get http://www.ikea.com/fr/fr/catalog/products/10288195/> (referer: http://www.ikea.com/fr/fr/catalog/productsaz/6/) 2016-11-08 11:16:45 [scrapy] debug: file (uptodate): downloaded image <get http://www.ikea.com/fr/fr/images/products/gunnern-armoire-verrouillable-rouge__0275675_pe413909_s4.jpg> referred in <none> error 1064: have error in sql syntax; check manual corresponds mysql server version right syntax use near '), 'meubles à miroir', '32', '10.4', '32', 0, 0)' @ line 2 2016-11-08 11:16:45 [scrapy] debug: scraped <200 http://www.ikea.com/fr/fr/catalog/products/10288195/> {'brand': 'ikea', 'couleur': [], 'description': u'armoire verrouillable', 'diametre': 0.0, 'gamme': u'meubles \xe0 miroir', 'hauteur': u'32', 'image_paths': 'images', 'image_urls': [u'http://www.ikea.com/fr/fr/images/products/gunnern-armoire-verrouillable-rouge__0275675_pe413909_s4.jpg'], 'images': [{'checksum': 'c95f7095c0b6645b5780bff39b47e3d2', 'path': 'full/20c2afef8324d798baac384970e9efb4557f9f19.jpg', 'url': 'http://www.ikea.com/fr/fr/images/products/gunnern-armoire-verrouillable-rouge__0275675_pe413909_s4.jpg'}], 'largeur': u'32', 'link': 'http://www.ikea.com/fr/fr/catalog/products/10288195/', 'longueur': 0.0, 'price': u'25', 'product': u'gunnern', 'profondeur': u'10.4'}
i have 2 different error message, don't understand why. along whole scrapping process, appeared alternatively. can please me ?
thank you
Comments
Post a Comment