Why python throws an "ImportError: No module named linkextractor" even if the Scrapy module already installed?

Question

I want to install portia. It requires scrapy and slybot. I also installed virtualenv (as recommended) and executed virtualenv Portia to create a python virtual environment only for this project. Now it requires to start twisted server with twistd -n slyd, but then it throws an Error:

  File "/home/john/Downloads/portia-master/slybot/slybot/linkextractor/__init__.py", line 6, in <module>
    from .base import BaseLinkExtractor, ALLOWED_SCHEMES   File "/home/john/Downloads/portia-master/slybot/slybot/linkextractor/base.py", line 6, in <module>
    from scrapy.linkextractor import IGNORED_EXTENSIONS ImportError: No module named linkextractor

I have Ubuntu 12.04, installed: python 2.7, pip, python-scrapy, python-twisted

What could be here wrong? I didn't use pip and virtualenv before, does seem something wrong with the deployment workflow (e.g. ENVIRONMENT variables, other virtualenv/pip configurations) I described above?

Answer 1

In python3, just add s to the end of linkextractor.

Replace :

from scrapy.linkextractor import LinkExtractor

with :

from scrapy.linkextractors import LinkExtractor
                         ^

and enjoy it.

for more information read the Scrapy documentation from here.

Answer 2

It misses out the middle part; contrib, of which linkextractor is a submodule.

It is supposed to be:

scrapy.contrib.linkextractor
       ^^^^^^^

Answer 3

Well, I think that you should see this issue (https://github.com/scrapinghub/portia/issues/13) in github scrapinhub repo.

I solved some errors with imports installing the libraries in my system instead of in virtualenv.

In my case, the libraries were jsonschema (pip install jsonschema), and slybot (pip install -e slybot), and after I ran requirements.txt.

Best regards!