ModuleNotFoundError: folder structure problem in my scrapy project?

Question

I am new to scrapy and vscode, and my project was working perfectly fine until I decided to get tidy with the folders before uploading on github . After that, whole project is not working anymore. I am pretty sure I messed up the folder structure:

└── real_estate/
    ├── project1/
    │   ├── project1_scrapy/
    │   │   ├── spiders/
    │   │   │   ├── __init__.py
    │   │   │   └── project1__spider.py
    │   │   ├── items.py
    │   │   ├── middlewares.py
    │   │   ├── pipelines.py
    │   │   └── settings.py
    │   └── scrapy.cfg
    └── project2/
        ├── project2_scrapy/
        │   ├── spiders/
        │   │   ├── __init__.py
        │   │   └── project2__spider.py
        │   ├── items.py
        │   ├── middlewares.py
        │   ├── pipelines.py
        │   └── settings.py
        └── scrapy.cfg

I am running to crawler on the folder of scrapy.cfg. Still getting the following error:

 File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'project1'

Answer 1

You can change the SPIDER_MODULES of project1/project1_scrapy/settings.py to make the scrapy search for the correct directory for the spider

https://docs.scrapy.org/en/latest/topics/settings.html#spider-modules

For your case,

SPIDER_MODULES = ["project1_scrapy.spiders"]

Answer 2

You are getting ModuleNotFoundError: No module named 'project1' because you can't change/rename the project that once you've created by the command scrapy startproject that would be

project1
 project1

but in your case.it's

project1
    project1_scrapy

It seems to be clear that you have renamed/added _scrapy with project1 that's why scrapy can't find the project1 and shows the mentioned error.

If you go to your project's settings.py file then you can see your project name like :

BOT_NAME = 'project1'

SPIDER_MODULES = ['project1.spiders']
NEWSPIDER_MODULE = 'project1.spiders'

So remove the _scrapy from project1_scrapy or create new project and never change/rename the project or make correct spider module name from settings.fy file according to the project folder. If you change something like bot name/spider modules You also need to change that portion from settings.py file Like You have changed the module name from your project folder project1 to project1_scrapy so you also have to change SPIDER_MODULES = ['project1_scrapy.spiders']

If you don't change anything then, Your project folders structure in vscode would be as above screenshot