Scrapy/Python - Attempting to stagger spiders

Question

Trying to stagger two spiders:

spider1 crawls and builds a list of URLs into a .csv

spider2 crawls from the .csv to then pull specific data

I keep getting this error: with open('urls.csv') as file: FileNotFoundError: [Errno 2] No such file or directory: 'urls.csv'

It looks like spider1 isn't able to fire first, and/or that python is checking for the file urls.csv because of the order of the code and is erroring out because the file doesn't exist yet.

This is the piece to stagger the crawls - it's something I grabbed from gitHub a while back, but the link appears to no longer be up. I've tried placing this in different spots, and even duplicating or splitting it up.

def crawl():
    yield runner.crawl(spider1)
    yield runner.crawl(spider2)
    reactor.stop()
crawl()
reactor.run()

I like to have urls.csv to troubleshoot the URLs, but it may be best to store the URLs in a list as a variable [though I haven't figured out the syntax to be able to do that yet]

Below is the full code I'm using. Any input would be deeply appreciated. Thank you!

import scrapy
from twisted.internet import reactor, defer
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

configure_logging()
settings = get_project_settings()
runner = CrawlerRunner(settings)

class spider1(scrapy.Spider):
    name = 'spider1'
    start_urls = [
        'https://tsd-careers.hii.com/en-US/search?keywords=alion&location='
    ]

    custom_settings = {'FEEDS': {r'urls.csv': {'format': 'csv', 'item_export_kwargs': {'include_headers_line': False,}, 'overwrite': True,}}}

    def parse(self, response):
        for job in response.xpath('//@href').getall():
            yield {'url': response.urljoin(job),}

        next_page = response.xpath('//a[@class="next-page-caret"]/@href').get()
        if next_page is not None:
            yield response.follow(next_page, callback=self.parse)


class spider2(scrapy.Spider):
    name = 'spider2'
    with open('urls.csv') as file:
        start_urls = [line.strip() for line in file]

    custom_settings = {'FEEDS': {r'data_tsdfront.xml': {'format': 'xml', 'overwrite': True}}}

    def parse(self, response):
        reqid = response.xpath('//li[6]/div/div[@class="secondary-text-color"]/text()').getall()
        yield {
            'reqid': reqid,
        }

@defer.inlineCallbacks
def crawl():
    yield runner.crawl(spider1)
    yield runner.crawl(spider2)
    reactor.stop()
crawl()
reactor.run()

Answer 1

I've come to understand that to use a variable would require a lot of restructuring.

I've done some revising after some more research and experimenting, and this may be ugly but I have everything working as desired. I can improve and restructure as my knowledge and experience grows.

import scrapy
from twisted.internet import reactor, defer
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
import pandas as pd

configure_logging()
settings = get_project_settings()
runner = CrawlerRunner(settings)

class spider1(scrapy.Spider):   
    name = 'spider1'    
    start_urls = [      
            'https://tsd-careers.hii.com/en-US/search?keywords=alion&location=' 
    ]   

    custom_settings = {'FEEDS': {r'urls.csv': {'format': 'csv', 'overwrite': True,}}}   

    def parse(self, response):
        for job in response.xpath('//@href').getall():
            yield {'url': response.urljoin(job),}       

        next_page = response.xpath('//a[@class="next-page-caret"]/@href').get()
        if next_page is not None:
            yield response.follow(next_page, callback=self.parse)

def read_csv(): 
    df = pd.read_csv('urls.csv')
    return df['url'].values.tolist()

class spider2(scrapy.Spider):   
    name = 'spider2'    
    def start_requests(self):
        for url in read_csv():
        yield scrapy.Request(url)   

custom_settings = {'FEEDS': {r'data_tsdfront.xml': {'format': 'xml', 'overwrite': True}}}   

    def parse(self, response):
        data = response.css('*').getall()
            yield {
            'data': data,   
            }

@defer.inlineCallbacks
def crawl():    
    yield runner.crawl(spider1)
    yield runner.crawl(spider2)
    reactor.stop()
crawl()
reactor.run()