Category "web-crawler"

Which parse method scrapy used to parse start_urls

I want scrapy to scrape some start urls and then follow the links in those pages according to rules. My spider is inherited from CrawlSpider and has start_urls

How to crawl question and answer of Google People Also Ask with Selenium and Python?

I used this code for crawl question and anwser of Google People Also Ask. I want use that for create idea for writer. But I can't get exactly that element, in t