'Scrapy on M1 Mac: MemoryError: Cannot allocate write+execute memory for ffi.callback()

I'm new to scrapy, and recently started using it on the M1 MacBook Air. I've encountered an issue.

For example, when I try to do something like this:

scrapy shell bbc.com

It would return me: MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks

It is worth noting that if I do the exact same thing on Windows, everything works. Also, when I use scrapy shell on websites like http://quotes.toscrape.com/ or HTTP websites on the M1 Mac, it also works. Seems like a HTTPS issue but I'm really not sure.

Any help will be greatly appreciated.

scrapy


Solution 1:[1]

What you need to do is just to upgrade requests module

pip3 install --upgrade requests

Solution 2:[2]

I've been struggling with this for a few months. For me, it seem so be a problem with pyopenssl. I have not found a solution.

If you try with an http website (instead of https), you may find that it works: scrapy shell http://httpforever.com

Relevant issue: https://github.com/pyca/pyopenssl/issues/873

Additional conversation: https://github.com/Azure/azure-cli/issues/16907

Update 20211129:

HTTPS sites such as scrapy shell https://www.bbc.com are working for me again. My environment:

  • M1 Macbook Pro
  • MacOS 12.0.1
  • Python 3.10.0
  • Scrapy==2.5.1

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Superneat
Solution 2

Related Questions