'Parse methods are not beign triggered

Description

The parse or any other method that is used to parse the HTML step-by-step are not beign triggered after the Request and there is no error in the logs.

Steps to Reproduce

  1. Assign the list of URLs to the start_urls class attribute
  2. Schedule the spider in Scrapyd
  3. The spider finishes before trigger the parse method

Expected behavior: Expect that all chainned parsing methods be triggered

Actual behavior: No method is triggered

Reproduces how often: All the time, it just doesn't work

Versions

Scrapy       : 2.5.0
lxml         : 4.7.1.0
libxml2      : 2.9.12
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 21.7.0
Python       : 3.10.2 (main, Jan 18 2022, 20:46:06) [GCC 10.3.1 20211027]
pyOpenSSL    : 21.0.0 (OpenSSL 1.1.1m  14 Dec 2021)
cryptography : 36.0.1
Platform     : Linux-5.13.0-27-generic-x86_64-with

Additional context

Here is an example of my code:

class MySpider(Spider):
    name = 'myspider'

    def __init__(self, *args, **kwargs):
        super(MySpider, self).__init__(*args, **kwargs)
        self.config = ConfigSchema().loads(self.config) // this object comes from rabbitmq
        self.start_urls = self.config.get('start_urls') // this returns a list of urls

    def parse(self, response):
        // ... do parse things, but it is not triggered
        return loader.load_item()

I hope someone could help me, thank you in advance!



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source