Category "scrapy"

Scrapy: How to output items in a specific json format

I output the scraped data in json format. Default scrapy exporter outputs list of dict in json format. Item type looks like: [{"Product Name":"Product1", "Cate

Get all link text and href in a page using scrapy

class LinkSpider(scrapy.Spider): name = "link" def start_requests(self): urlBasang = "https://bloomberg.com" yield scrapy.Request(url =

Which parse method scrapy used to parse start_urls

I want scrapy to scrape some start urls and then follow the links in those pages according to rules. My spider is inherited from CrawlSpider and has start_urls