How can I redirect module imports with modern Python?

Question

I am maintaining a python package in which I did some restructuring. Now, I want to support clients who still do from my_package.old_subpackage.foo import Foo instead of the new from my_package.new_subpackage.foo import Foo, without explicitly reintroducing many files that do the forwarding. (old_subpackage still exists, but no longer contains foo.py.)

I have learned that there are "loaders" and "finders", and my impression was that I should implement a loader for my purpose, but I only managed to implement a finder so far:

RENAMED_PACKAGES = {
    'my_package.old_subpackage.foo': 'my_package.new_subpackage.foo',
}

# TODO: ideally, we would not just implement a "finder", but also a "loader"
# (using the importlib.util.module_for_loader decorator); this would enable us
# to get module contents that also pass identity checks
class RenamedFinder:

    @classmethod
    def find_spec(cls, fullname, path, target=None):
        renamed = RENAMED_PACKAGES.get(fullname)
        if renamed is not None:
            sys.stderr.write(
                f'WARNING: {fullname} was renamed to {renamed}; please adapt import accordingly!\n')
            return importlib.util.find_spec(renamed)
        return None

sys.meta_path.append(RenamedFinder())

https://docs.python.org/3.5/library/importlib.html#importlib.util.module_for_loader and related functionality, however, seem to be deprecated. I know it's not a very pythonic thing I am trying to achieve, but I would be glad to learn that it's achievable.

Answer 1

On import of your package's __init__.py, you can place whatever objects you want into sys.modules, the values you put in there will be returned by import statements:

from . import new_package
from .new_package import module1, module2
import sys

sys.modules["my_lib.old_package"] = new_package
sys.modules["my_lib.old_package.module1"] = module1
sys.modules["my_lib.old_package.module2"] = module2

If someone now uses import my_lib.old_package or import my_lib.old_package.module1 they will obtain a reference to my_lib.new_package.module1. Since the import machinery already finds the keys in the sys.modules dictionary, it never even begins looking for the old files.

If you want to avoid importing all the submodules immediately, you can emulate a bit of lazy loading by placing a module with a __getattr__ in sys.modules:

from types import ModuleType
import importlib
import sys

class LazyModule(ModuleType):
 def __init__(self, name, mod_name):
  super().__init__(name)
  self.__mod_name = name

 def __getattr__(self, attr):
  if "_lazy_module" not in self.__dict__:
    self._lazy_module = importlib.import(self.__mod_name, package="my_lib")
  return self._lazy_module.__getattr__(attr)

sys.modules["my_lib.old_package"] = LazyModule("my_lib.old_package", "my_lib.new_package")

Answer 2

In the init file of the old module, have it import from the newer modules
Old (package.oldpkg):

foo = __import__("Path to new module")

New (package.newpkg):

class foo:
  bar = "thing"

so
package.oldpkg.foo.bar is the same as package.newpkg.foo.bar

Hope this helps!

Answer 3

I think that this is what you are looking for:

RENAMED_PACKAGES = {
    'my_package.old_subpackage.foo': 'my_package.new_subpackage.foo',
}

class RenamedFinder:

    @classmethod
    def find_spec(cls, fullname, path, target=None):
        renamed = RENAMED_PACKAGES.get(fullname)
        if renamed is not None:
            sys.stderr.write(
                f'WARNING: {fullname} was renamed to {renamed}; please adapt import accordingly!\n')
            spec = importlib.util.find_spec(renamed)
            spec.loader = cls
            return spec
        return None

    @staticmethod
    def create_module(spec):
        return importlib.import_module(spec.name)

    @staticmethod
    def exec_module(module):
        pass

sys.meta_path.append(RenamedFinder())

Still, IMO the approach that manipulates sys.modules is preferable as it is more readable, more explicit, and provides you much more control. It might become useful especially in further versions of your package when my_package.new_subpackage.foo starts to diverge from my_package.old_subpackage.foo while you would still need to provide the old one for backward compatibility. For that reason, you would maybe need to preserve the code of both anyway.

Answer 4

Consolidate all the old package names into my_package.
Old packages (old_package):

• image_processing (class) Will be deleted and replaced by better_image_processing
• text_recognition (class) Will be deleted and replaced by better_text_recognition
• foo (variable) Will be moved to better_text_recognition
• still_there (class) Will not move

New packages:

• super_image_processing
• better_text_recognition

Redirector (class of my_package):

class old_package:
   image_processing = super_image_processing # Will be replaced
   text_recognition = better_text_recognition # Will be replaced

Your main new module (my_package):

#imports here
class super_image_processing:
  def its(gets,even,better):
    pass
class better_text_recognition:
  def now(better,than,ever):
    pass
class old_package:
   #Links
   image_processing = super_image_processing
   text_recognition = better_text_recognition
   still_there = __import__("path to unchanged module")

This allows you to delete some files and keep the rest. If you want to redirect variables you would do:

class super_image_processing:
  def its(gets,even,better):
    pass
class better_text_recognition:
  def now(better,than,ever):
    pass
class old_package:
   #Links
   image_processing = super_image_processing
   text_recognition = better_text_recognition
   foo = text_recognition.foo
   still_there = __import__("path to unchanged module")

Would this work?