Python: perform relative import when using __import__?

Question

Here are the files in this test:

main.py
app/
 |- __init__.py
 |- master.py
 |- plugin/
 |-  |- __init__.py
 |-  |- p1.py
 |-  |_ p2.py

The idea is to have a plugin-capable app. New .py or .pyc files can be dropped into plugins that adhere to my API.

I have a master.py file at the app level that contains global variables and functions that any and all plugins may need access to, as well as the app itself. For the purposes of this test, the "app" consists of a test function in app/__init__.py. In practice the app would probably be moved to separate code file(s), but then I'd just use import master in that code file to bring in the reference to master.

Here's the file contents:

main.py:

import app

app.test()
app.test2()

app/__init__.py:

import sys, os

from plugin import p1

def test():
        print "__init__ in app is executing test"
        p1.test()

def test2():
        print "__init__ in app is executing test2"
        scriptDir = os.path.join ( os.path.dirname(os.path.abspath(__file__)), "plugin" )
        print "The scriptdir is %s" % scriptDir
        sys.path.insert(0,scriptDir)
        m = __import__("p2", globals(), locals(), [], -1)
        m.test()

app/master.py:

myVar = 0

app/plugin/__init__.py:

<empty file>

app/plugin/p1.py:

from .. import master

def test():
    print "test in p1 is running"
    print "from p1: myVar = %d" % master.myVar

app/plugin/p2.py:

from .. import master

def test():
    master.myVar = 2
    print "test in p2 is running"
    print "from p2, myVar: %d" % master.myVar

Since I explicitly import the p1 module, everything works as expected. However, when I use __import__ to import p2, I get the following error:

__init__ in app is executing test
test in p1 is running
from p1: myVar = 0
__init__ in app is executing test2
The scriptdir is ....../python/test1/app/plugin
Traceback (most recent call last):
  File "main.py", line 4, in <module>
    app.test2()
  File "....../python/test1/app/__init__.py", line 17, in test2
    m = __import__("p2", globals(), locals(), [], -1)
  File "....../python/test1/app/plugin/p2.py", line 1, in <module>
    from .. import master
ValueError: Attempted relative import in non-package

Execution proceeds all the way through the test() function and errors out right as test2() tries to execute its __import__ statement, which in turn p2 tries to do a relative import (which does work when p1 is imported explicitly via the import statement, recall)

It's clear that using __import__ is doing something different than using the import statement. The Python docs state that using import simply translates to an __import__ statement internally but there has to be more going on than meets the eye.

Since the app is plugin-based, coding explicit import statements in the main app would of course not be feasible. Using import itself within the

What am I missing here? How can I get Python to behave as expected when manually importing modules using __import__? It seems maybe I'm not fully understanding the idea of relative imports, or that I'm just missing something with respect to where the import is occurring (i.e. inside a function rather than at the root of the code file)

EDIT: I found the following possible, but unsuccessful solutions:

m = __import__("p2",globals(),locals(),"plugin")

(returns the same exact error as above)

m = __import__("plugin",fromlist="p2")

(returns a reference to app.plugin, not to app.plugin.p2)

m = __import__("plugin.p2",globals(),locals())

(returns a reference to app.plugin, not to app.plugin.p2)

import importlib
m = importlib.import_module("plugin.p2")

(returns:)

Traceback (most recent call last):
  File "main.py", line 4, in <module>
    app.test2()
  File "....../python/test1/app/__init__.py", line 20, in test2
    m = importlib.import_module("plugin.p2")
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named plugin.p2

Answer 1

I never did find a solution, so I ended up deciding to restructure the program.

What I did was set up the main app as a class. Then, I also changed each plugin into a class. Then, as I load plugins using import, I also instantiate the class inside each plugin which has a predefined name, and pass in the reference to the main app class.

This means that each class can directly read and manipulate variables back in the host class simply by using the reference. It is totally flexible because anything that the host class exports is accessible by all the plugins.

This turns out to be more effective and doesn't depend on relative paths and any of that stuff. It also means one Python interpreter could in theory run multiple instances of the host app simultaneously (on different threads for example) and the plugins will still refer back to the correct host instance.

Here's basically what I did:

main.py:

import os, os.path, sys

class MyApp:

    _plugins = []

    def __init__(self):
        self.myVar = 0

    def loadPlugins(self):
        scriptDir = os.path.join ( os.path.dirname(os.path.abspath(__file__)), "plugin" )   
        sys.path.insert(0,scriptDir)
        for plug in os.listdir(scriptDir):
            if (plug[-3:].lower() == ".py"):
                m = __import__(os.path.basename(plug)[:-3])
                self._plugins.append(m.Plugin(self))

    def runTests(self):
        for p in self._plugins:
            p.test()

if (__name__ == "__main__"):
    app = MyApp()
    app.loadPlugins()
    app.runTests()

plugin/p1.py:

class Plugin:

    def __init__(self, host):
        self.host = host

    def test(self):
        print "from p1: myVar = %d" % self.host.myVar

plugin/p2.py:

class Plugin:

    def __init__(self, host):
        self.host = host

    def test(self):
        print "from p2: variable set"
        self.host.myVar = 1
        print "from p2: myVar = %d" % self.host.myVar

There is some room to improve this, for example, validating each imported .py file to see if it's actually a plugin and so on. But this works as expected.

Answer 2

I've had a similar problem.
__import__ only imports submodules if all parent __init__.py files are empty. You should use importlib instead

import importlib

p2 = importlib.import_module('plugin.p2')

Answer 3

Have you tried the following syntax:

How to use python's import function properly __import__()

It worked for me with a similar problem...

Answer 4

I have managed to find a solution to the problem.
By taking your example the following static import is needed to be dynamic

from .plugin import p2

the "." near plugin means there is a need to relative import and not absolute import.

I was able to do that with the following code snipset:

plugin  = __import__('plugin', globals(), locals(), level=1, fromlist=['p2'])
p2      = getattr(plugin, 'p2')

level=1 Relative import parameter
fromlist Specify which sub modules to take from plugin module

As you mentioned, plugin holds the reference to 'plugin', thus additional getattr is needed to grep p2 from plugin