Issues with pip failing to install some modules (while successful with some others)…what else can I try, do I need a venv?

Question

I have Python 3.8.2 (latest) installed on a Big Sur macOS M1 Mac mini and it's installed into the Command Line Tools directory (/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework).

I've been attempting to load various python 3 packages/modules using various methods as recommended in the official docs, here on SO and elsewhere.

Some modules load and some fail to complete the installation (see below for example error details), whatever syntax I use to invoke the download and installation.

Should I attempt to work around these issue with a Virtual Environment (adding complication as I'm pretty new to Python)?

If the answer is "yes, you should be using a venv to solve this", should I use a high level tool like pipenv as recommended here in HHGtP or a lower level tool like venv or virtualenv as recommended in the offical docs?

---

Background:

Modules which successfully installed and I can import in the REPL:

• numpy. ✓
• openpyxl ✓ <— Installed into my user folder
• pyexcel ✓ <— In user folder
• superpower ✓
• urllib3 ✓
• xlrd ✓
• xlwt ✓

Modules that failed to install:

• matplotlib ❌ imports in python3 REPL though so seems to be good

• mpl_toolkits ❌ imports in python3 REPL though so seems to be good

• netCDF4 ❌

• opt_debug ❌ Could not find a version that satisfies the requirement opt_debug (from versions: none)

• pylab ❌ Could not find a version that satisfies

• pyproj ❌

• PyQt5 ❌

• shutil ❌

I've downloaded a Python project from GitHub called SIREN that I am attempting to build for macOS (with the assistance of the author of SIREN who uses Linux and Windows but not macOS). It has a number of module dependencies outside Python 3's standard library, in addition to many of its own modules in the project files (the project modules load ok in the Python 3 REPL but many external dependancies don't).

Almost always the modules attempt to install in my user folder, not in the CLT base installation of Python due to a write protect error in the install process. I know it's recommended not to, but in desperation I even attempted to use $sudo python3 -m pip install PyQt5 and although it spend a few minutes attempting to install it ultimately failed with some kind of metadata error. (see below).

These commands indicate I have the tools installed and path is operating well enough for them to operate:

$ python3 --version

Python 3.8.2

$ python3 -m pip --version

pip 22.0.4 from /Users/me/Library/Python/3.8/lib/python/site-packages/pip (python 3.8)

Here's some examples of failure errors:

$ pip install PyQt5
Defaulting to user installation because normal site-packages is not writeable Collecting PyQt5
Using cached PyQt5-5.15.6.tar.gz (3.2 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error
   
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [29 lines of output]
    Traceback (most recent call last):
     File "/Users/alastairleith/Library/Python/3.8/lib/python/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 156, in prepare_metadata_for_build_wheel
      hook = backend.prepare_metadata_for_build_wheel
  AttributeError: module 'sipbuild.api' has no attribute 'prepare_metadata_for_build_wheel'
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/Users/alastairleith/Library/Python/3.8/lib/python/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/Users/alastairleith/Library/Python/3.8/lib/python/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/alastairleith/Library/Python/3.8/lib/python/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 160, in prepare_metadata_for_build_wheel
      whl_basename = backend.build_wheel(metadata_directory, config_settings)
    File "/private/var/folders/f3/0sr9wxg16l3dbxncx0fjdz6m0000gn/T/pip-build-env-51w_hx87/overlay/lib/python3.8/site-packages/sipbuild/api.py", line 51, in build_wheel
      project = AbstractProject.bootstrap('pep517')
    File "/private/var/folders/f3/0sr9wxg16l3dbxncx0fjdz6m0000gn/T/pip-build-env-51w_hx87/overlay/lib/python3.8/site-packages/sipbuild/abstract_project.py", line 83, in bootstrap
      project.setup(pyproject, tool, tool_description)
    File "/private/var/folders/f3/0sr9wxg16l3dbxncx0fjdz6m0000gn/T/pip-build-env-51w_hx87/overlay/lib/python3.8/site-packages/sipbuild/project.py", line 594, in setup
      self.apply_user_defaults(tool)
    File "project.py", line 63, in apply_user_defaults
      super().apply_user_defaults(tool)
    File "/private/var/folders/f3/0sr9wxg16l3dbxncx0fjdz6m0000gn/T/pip-build-env-51w_hx87/overlay/lib/python3.8/site-packages/pyqtbuild/project.py", line 70, in apply_user_defaults
      super().apply_user_defaults(tool)
    File "/private/var/folders/f3/0sr9wxg16l3dbxncx0fjdz6m0000gn/T/pip-build-env-51w_hx87/overlay/lib/python3.8/site-packages/sipbuild/project.py", line 241, in apply_user_defaults
      self.builder.apply_user_defaults(tool)
    File "/private/var/folders/f3/0sr9wxg16l3dbxncx0fjdz6m0000gn/T/pip-build-env-51w_hx87/overlay/lib/python3.8/site-packages/pyqtbuild/builder.py", line 67, in apply_user_defaults
      raise PyProjectOptionException('qmake',
  sipbuild.pyproject.PyProjectOptionException
  [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  error: metadata-generation-failed

  × Encountered error while generating package metadata.
   ╰─> See above for output.

  note: This is an issue with the package mentioned above, not pip.
  hint: See above for details.

When pip3 fails to install in one instance and when I check pip3 --version I get told to read this note Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue. it warns against using sudo so I avoided it for some time yesterday but eventually gave in to temptation. It ran the install process for PyQt5 for a few minutes and did lots of downloading but the installation failed also. I’m not sure if it tried to install in the home directory of python 3 or in my user folder, don't recall. I tried to uninstall after it failed.