How to extract dependencies information from a setup.py

Question

I have a python project, let's call it foobar, there is a setup.py script in project root directory like all Python projects. For example

• foobar
  • setup.py

setup.py file content:

from ez_setup import use_setuptools
use_setuptools()

from setuptools import setup, find_packages
setup(
    name='foobar',
    version='0.0.0',
    packages=find_packages(),
    install_requires=[
        'spam==1.2.3',
        'eggs>=4.5.6',
    ],
)

I need to get dependencies information from that setup.py file using Python. The part I want would be

[
    'spam==1.2.3',
    'eggs>=4.5.6',
]

in the example above. I don't want to install this package, all I need is the dependencies information. Certainly, I can use regular expression to parse it, but that would be ugly, I can also use Python AST to parse it, but I think there should already be some tool can do this. What is the best way to do so?

Answer 1

You can use distutils.core's run_setup:

from distutils.core import run_setup

result = run_setup("./setup.py", stop_after="init")
result.install_requires
['spam==1.2.3', 'eggs>=4.5.6']

This way there is no need to mock anything and you can potentially extract more information about the project than would be possible by mocking the setup() call.

Note that this solution might be problematic as there apparently is active work being done to deprecate distutils. See comments for details.

Answer 2

It seems to me that you could use mock to do the work (assuming that you have it installed and that you have all the setup.py requirements...). The idea here is to just mock out setuptools.setup and inspect what arguments it was called with. Of course, you wouldn't really need mock to do this -- You could monkey patch setuptools directly if you wanted ...

import mock  # or `from unittest import mock` for python3.3+.
import setuptools

with mock.patch.object(setuptools, 'setup') as mock_setup:
    import setup  # This is setup.py which calls setuptools.setup

# called arguments are in `mock_setup.call_args`
args, kwargs = mock_setup.call_args
print kwargs.get('install_requires', [])

Answer 3

Pretty similar idea to @mgilson's solution, I use ast, parse setup.py module, insert a mock setup method before setup call, and collect the args and kwargs.

import ast
import textwrap


def parse_setup(setup_filename):
    """Parse setup.py and return args and keywords args to its setup
    function call

    """
    mock_setup = textwrap.dedent('''\
    def setup(*args, **kwargs):
        __setup_calls__.append((args, kwargs))
    ''')
    parsed_mock_setup = ast.parse(mock_setup, filename=setup_filename)
    with open(setup_filename, 'rt') as setup_file:
        parsed = ast.parse(setup_file.read())
        for index, node in enumerate(parsed.body[:]):
            if (
                not isinstance(node, ast.Expr) or
                not isinstance(node.value, ast.Call) or
                node.value.func.id != 'setup'
            ):
                continue
            parsed.body[index:index] = parsed_mock_setup.body
            break

    fixed = ast.fix_missing_locations(parsed)
    codeobj = compile(fixed, setup_filename, 'exec')
    local_vars = {}
    global_vars = {'__setup_calls__': []}
    exec(codeobj, global_vars, local_vars)
    return global_vars['__setup_calls__'][0]

Answer 4

Great answers here already, but I had to modify the answer from @mgilson just a bit to get it to work for me because there are (apparently) incorrectly configured projects in my source tree, and the wrong setup was being imported. In my solution I am temporarily creating a copy of the setup.py file by another name so that I can import it and the mock can intercept the correct install_requires data.

import sys 
import mock
import setuptools
import tempfile
import os

def import_and_extract(parent_dir):
    sys.path.insert(0, parent_dir)
    with tempfile.NamedTemporaryFile(prefix="setup_temp_", mode='w', dir=parent_dir, suffix='.py') as temp_fh:
        with open(os.path.join(parent_dir, "setup.py"), 'r') as setup_fh:
            temp_fh.write(setup_fh.read()) 
            temp_fh.flush()
        try:
            with mock.patch.object(setuptools, 'setup') as mock_setup:
                module_name = os.path.basename(temp_fh.name).split(".")[0]
                __import__(module_name)
        finally:
            # need to blow away the pyc
            try:
                os.remove("%sc"%temp_fh.name)
            except:
                print >> sys.stderr, ("Failed to remove %sc"%temp_fh.name)
        args, kwargs = mock_setup.call_args
        return sorted(kwargs.get('install_requires', []))


if __name__ == "__main__":
    if len(sys.argv) > 1:
        thedir = sys.argv[1]
        if not os.path.isdir(thedir):
            thedir = os.path.dirname(thedir)
        for d in import_and_extract(thedir):
            print d
    else:   
        print >> sys.stderr, ("syntax: %s directory"%sys.argv[0])

Answer 5

If you need only to parse setup.cfg, you can use the configparser module in the standard library:

>>> import configparser
>>> config = configparser.ConfigParser()
>>> config.read('setup.cfg')
['setup.cfg']
>>> config.sections()
['metadata', 'options', 'options.extras_require', 'options.package_data', 'options.packages.find']
>>> config['options']['install_requires']
'pandas'

Answer 6

Using run_setup from distutils.core is great, but some setup.py files do some additional actions before running setup(...), so parsing the code seems the only solution.

import ast
import sys

path = sys.argv[1]
parsed = ast.parse(open(path).read())
for node in parsed.body:
    if not isinstance(node, ast.Expr):
        continue
    if not isinstance(node.value, ast.Call):
        continue
    if node.value.func.id != "setup":
        continue
    for keyword in node.value.keywords:
        if keyword.arg == "install_requires":
            requirements = ast.literal_eval(keyword.value)
            print("\n".join(requirements))