python - Scrapy CrawlSpider doesn't crawl the first landing page -


i new scrapy , working on scraping exercise , using crawlspider. although scrapy framework works beautifully , follows relevant links, can't seem make crawlspider scrape first link (the home page / landing page). instead goes directly scrape links determined rule doesn't scrape landing page on links are. don't know how fix since not recommended overwrite parse method crawlspider. modifying follow=true/false doesn't yield results. here snippet of code:

class downloadspider(crawlspider):     name = 'downloader'     allowed_domains = ['bnt-chemicals.de']     start_urls = [         "http://www.bnt-chemicals.de"                 ]     rules = (            rule(sgmllinkextractor(aloow='prod'), callback='parse_item', follow=true),         )     fname = 1      def parse_item(self, response):         open(str(self.fname)+ '.txt', 'a').write(response.url)         open(str(self.fname)+ '.txt', 'a').write(','+ str(response.meta['depth']))         open(str(self.fname)+ '.txt', 'a').write('\n')         open(str(self.fname)+ '.txt', 'a').write(response.body)         open(str(self.fname)+ '.txt', 'a').write('\n')         self.fname = self.fname + 1

just change callback parse_start_url , override it:

from scrapy.contrib.spiders import crawlspider, rule scrapy.contrib.linkextractors.sgml import sgmllinkextractor  class downloadspider(crawlspider):     name = 'downloader'     allowed_domains = ['bnt-chemicals.de']     start_urls = [         "http://www.bnt-chemicals.de",     ]     rules = (         rule(sgmllinkextractor(allow='prod'), callback='parse_start_url', follow=true),     )     fname = 0      def parse_start_url(self, response):         self.fname += 1         fname = '%s.txt' % self.fname          open(fname, 'w') f:             f.write('%s, %s\n' % (response.url, response.meta.get('depth', 0)))             f.write('%s\n' % response.body)

Comments

Post a Comment

Popular posts from this blog

asp.net mvc 3 - Using mvc3, I need to add a username/password to the sql connection string at runtime -

i see how build string using stringbuilders (sql/entity framework); do cause connection string used in dbcontext? example context: public class pophistorydbcontext : dbcontext { public dbset<pophistory> histories {get;set} } public class pophistoryrepository: ihistoryrepository { private pophistorydbcontext context = new pophistorydbcontext(); public iqueryable<pophistory> histories{ {return context.histories;} } } example usage in controller: public class historycontroller : controller { private ihistoryrepository repository = new pophistoryrepository(); var histories = h in repository.histories (h.modifytimestamp > today) select h; if (!string.isnullorempty(searchstring)) { histories = histories.where(h => h.dn.toupper().contains(searchstring.toupper())); } thank you
Read more

kineticjs - draw multiple lines and delete individual line -

i can draw multiple lines on canvas; added layer using drawscene dlayera1.add(line); line.getpoints()[0].x = mousepos.x; line.getpoints()[0].y = mousepos.y; line.getpoints()[1].x = mousepos.x; line.getpoints()[1].y = mousepos.y; moving = true; dlayera1.drawscene(); http://jsfiddle.net/user373721/xzead/1/ . is there way delete individual line? currently, you'll have modify points array this: line.getpoints()[0].splice(index, 1); since arrays modified reference, modifying returned array modifies "source of truth" points array attr
Read more

thumbnails - jQuery image rotate on hover -

any chance change image src on hover , count 10 , again first? this need, default thumb, flv-3.jpg ! <img src="/thumbs/e/b/d/6/d/ebd6d2d6d99e/ebd6d2d6d99e.flv-3.jpg" /> on mouse hover need change number flv-1.jpg flv-2.jpg flv-3.jpg flv-4.jpg etc.. up 10, , again , start number 1, when move mouse thumbs need default thumb flv-3.jpg this im use not solutions, var _thumbs = new array(); function changethumb(index, i, num_thumbs, path) { if (_thumbs[index]) { document.getelementbyid(index).src = path + + ".jpg"; preload = new image(); preload_image_id = (i + 1 > num_thumbs) ? 1 : + 1; preload.src = path + preload_image_id + ".jpg"; = % num_thumbs; i++; settimeout("changethumb('" + index + "'," + + ", " + num_thumbs + ", '" + path + "')", 800) } } function startthumbchange(index, nu
Read more