PyInstaller not able to create exe using geopandas and fiona. Even if I create with some changes in .spec file exe is not working

Question

I am using simple code to create exe using pyinstaller using geopandas and fiona as imports.

Sample code:

import glob
import geopandas as gpd
from pyproj import _datadir, datadir
import fiona
from osgeo import gdal, ogr, osr
from fiona.ogrext import Iterator, ItemsIterator, KeysIterator
from geopandas import GeoDataFrame
print("Hello")

I was not able to create exe using PyInstaller for this sample code as geopandas caused issues. I did some changes in .spec file as per one of the posts here. This allowed me to create exe somehow using following .spec file content:

block_cipher = None

import os
from PyInstaller.utils.hooks import collect_data_files # this is very helpful
from osgeo import gdal, ogr, osr
from fiona.ogrext import Iterator, ItemsIterator, KeysIterator
from geopandas import GeoDataFrame
rTreeDlls = 'C:\\Users\\supadhayay\\AppData\\Local\\Programs\\Python\\Python37\\Lib\\site-packages\\rtree'

paths = [
    'C:\\Users\\supadhayay',
    rTreeDlls,
    'C:\\Users\\supadhayay\\AppData\\Local\\Programs\\Python\\Python37\\DLLs',
    'C:\\Users\\supadhayay\\AppData\\Local\\Programs\\Python\\Python37\\Lib\\site-packages\\osgeo'
]

_osgeo_pyds = collect_data_files('osgeo', include_py_files=True)
_osgeo_pyds  = _osgeo_pyds  + collect_data_files('fiona', include_py_files=True)

osgeo_pyds = []
for p, lib in _osgeo_pyds:
    if '.pyd' in p or '.pyx' in p or '.pyc' in p:
        osgeo_pyds.append((p, '.'))

print(osgeo_pyds)

binaries = osgeo_pyds +[
    (os.path.join(rTreeDlls,'spatialindex-64.dll'), '.'),
    (os.path.join(rTreeDlls,'spatialindex_c.dll'),'.'),
]

hidden_imports = [
    'fiona',
    'gdal',
    'shapely',
    'shapely.geometry',
    'pyproj',
    'rtree',
    'geopandas.datasets',
    'pytest',
    'pandas._libs.tslibs.timedeltas',
]


a = Analysis(['D:\\SDR\\Repo\\Main\\DBEngg\\Spatial Data Repository\\States_Data_Processing_With_Geometry\\States_Data_Processing_With_Geometry_MP.py'],
             pathex=paths,
             binaries=osgeo_pyds +[('C:\\Users\\supadhayay\\AppData\\Local\\Programs\\Python\\Python37\\Lib\\site-packages\\shapely\\DLLs\\geos_c.dll', '.'),('C:\\Users\\supadhayay\\AppData\\Local\\Programs\\Python\\Python37\\Lib\\site-packages\\rtree\\spatialindex_c.dll', '.'), ('C:\\Users\\supadhayay\\AppData\\Local\\Programs\\Python\\Python37\\Lib\\site-packages\\rtree\\spatialindex-64.dll', '.')],
             datas=collect_data_files('geopandas', subdir='datasets') + [('D:\\SDR\\Repo\\Main\\DBEngg\\Spatial Data Repository\\States_Data_Processing_With_Geometry\\lg-logo-rms.png','.'),('D:\\SDR\\Repo\\Main\\DBEngg\\Spatial Data Repository\\States_Data_Processing_With_Geometry\\SQL_States_Data_Processing.sql','.')],
             hiddenimports=hidden_imports,
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          a.binaries,
          a.zipfiles,
          a.datas,
          [],
          name='States_Data_Processing_With_Geometry_MP',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          runtime_tmpdir=None,
          console=True )

After exe is getting created, I am not able to execute exe as it gives following error:

File "fiona\ogrext.pyx", line 1, in init fiona.ogrext ModuleNotFoundError: No module named 'fiona._shim'

I installed fiona using https://www.lfd.uci.edu/~gohlke/pythonlibs version: Fiona‑1.8.6‑cp37‑cp37m‑win_amd64.whl I can see _shim file available in fiona folder in sitepackages. Please help

Answer 1

I fixed it myself by adding 'fiona._shim' in hidden_imports in .spec file.

Answer 2

This and similar errors can be addressed by passing appropriate arguments to pyinstaller. For a script file called script.py, you want something like this:

pyinstaller --hidden-import fiona._shim --hidden-import fiona.schema script.py

Which packages need the --hidden-import treatment can be read directly from the ModuleNotFoundError. See also this portion of the PyInstaller documentation.

(I think this is what @suryakant-upadhayay meant by his self-answer, but I've also constructed a working case for my own project using a few additional flags that may not be related to geopandas.)

In general, you can also add community-defined "hooks" which should make PyInstaller better complex packages. See for example, this pull request