'Parse methods are not beign triggered
Description
The parse or any other method that is used to parse the HTML step-by-step are not beign triggered after the Request and there is no error in the logs.
Steps to Reproduce
- Assign the list of URLs to the
start_urlsclass attribute - Schedule the spider in Scrapyd
- The spider finishes before trigger the
parsemethod
Expected behavior: Expect that all chainned parsing methods be triggered
Actual behavior: No method is triggered
Reproduces how often: All the time, it just doesn't work
Versions
Scrapy : 2.5.0
lxml : 4.7.1.0
libxml2 : 2.9.12
cssselect : 1.1.0
parsel : 1.6.0
w3lib : 1.22.0
Twisted : 21.7.0
Python : 3.10.2 (main, Jan 18 2022, 20:46:06) [GCC 10.3.1 20211027]
pyOpenSSL : 21.0.0 (OpenSSL 1.1.1m 14 Dec 2021)
cryptography : 36.0.1
Platform : Linux-5.13.0-27-generic-x86_64-with
Additional context
Here is an example of my code:
class MySpider(Spider):
name = 'myspider'
def __init__(self, *args, **kwargs):
super(MySpider, self).__init__(*args, **kwargs)
self.config = ConfigSchema().loads(self.config) // this object comes from rabbitmq
self.start_urls = self.config.get('start_urls') // this returns a list of urls
def parse(self, response):
// ... do parse things, but it is not triggered
return loader.load_item()
I hope someone could help me, thank you in advance!
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
