Python config ImportError

Question

I am working on a python project and whose directory structure looks like this,

SEC-Edgar
├── SECEdgar
│   ├── __init__.py
│   ├── companylist.txt
│   ├── crawler.py
│   ├── crawler.pyc
│   ├── data.txt
│   └── test.py
├── config.py
├── requirements.txt
└── setup.py

I am trying to use config module inside crawler.py but it is giving an ImportError.

Traceback (most recent call last):
  File "SECEdgar/test.py", line 3, in <module>
    from crawler import SecCrawler
  File "/Users/rahul/Code/SEC-Edgar/SECEdgar/crawler.py", line 9, in <module>
    from config import DEFAULT_DATA_PATH
ImportError: No module named config

The import statement in crawler.py is

 from config import DEFAULT_DATA_PATH

I am not able to understand how import works in python, particularly when it comes to different directories like importing from root to base directory.

Do I need to add __init__.py on root directory also, so that it becomes a package and then use . to import it?

Is there a better way to handle import or am I missing some fundamentals here?

Answer 1

use from ..config import DEFAULT_DATA_PATH

Refer: Intra-package References

If you get error : Attempted relative import in non-package

Refer : this question

Answer 2

First change the from config import DEFAULT_DATA_PATH to from ..config import DEFAULT_DATA_PATH then add an empty __init__.py in SEC-Edgar directory.

Then, try executing your module as python -m SEC-Edgar.SECEdgar.test or like that for any module you are running, from the location(directory) where SEC-Edgar exists.

There are three things to note here.

1. your config.py and crawler.py modules doesn't share the same parent directory. So, when you try from config import DEFAULT_DATA_PATH from crawler.py Python can't import it for you.
2. We can import modules(.py files) from other directories in python only if that directory is a package. It can be done by just adding an empty __init__.py file in that directory.
3. We can now use relative imports. But, During relative imports Python uses the __name__ property of the module, As per PEP 328:

Relative imports use a module's name attribute to determine that module's position in the package hierarchy. If the module's name does not contain any package information (e.g. it is set to 'main') then relative imports are resolved as if the module were a top level module, regardless of where the module is actually located on the file system.

So, we running it with -m option to supply the package informations for Python to perform relative imports.

[In response to comment] Here is what working for me,

SEC-Edgar
??? SECEdgar
?   ??? __init__.py
?   ??? crawler.py
?   ??? test.py
??? __init__.py
??? config.py

I'm outside SEC-Edgar directory,

C:\Users\username\Documents> ls
SEC-EDGAR  OTHERS

C:\Users\username\Documents> python -m SEC-Edgar.SECEdgar.test
Lets assume I'm an SecCrawler

config.py

DEFAULT_DATA_PATH = "some/path"

crawler.py

from ..config import DEFAULT_DATA_PATH

SecCrawler = "Lets assume I'm an SecCrawler"

test.py

from crawler import SecCrawler

print(SecCrawler)

Answer 3

Your code is fine, the issue is caused by Python path.

try to answer this question: how/where can python find the module called 'config'?

So, I suggest you find these steps to understand and solve the issue:

1. reduce the scope in your crawler.py, use this main:

   if __name__ == '__main__':
       import sys
       print(sys.path)
   
       from config import DEFAULT_DATA_PATH
       print 'DEFAULT_DATA_PATH=%s' % DEFAULT_DATA_PATH
   

2. print(sys.path) will show you the python path. you probally cannot find your project root in the path.
3. if you're clear that you need to manage the python path, you may config it in your IDE. e.g. I'm using pycharm, I'll select the source root in settings/Project Structure and check 'Add source roots to PYTHONPATH' in Run/Debug Configurations.
4. after you configured it, re-run your crawler.py, the output will be samilar as this: ['C:\\dev\\projects\\SEC-Edgar\\SECEdgar', ... 'C:\\dev\\projects\\SEC-Edgar'.] DEFAULT_DATA_PATH=TEST TEST will be the value in SEC-Edgar\config.py

this is only for dev, for packaing/deployment, you'll need to config your setup.py properly. you may refer to: https://docs.python.org/2/distutils/setupscript.html