How to mock os.listdir to pretend files and directories in Python?

Question

I have a proprietary repository format and I'm trying to develop a Python module to process these repositories. Repo format goes as:

/home/X/
       |
       + alpha/
       |
       + beta/
       |
       + project.conf

Here, X is a project. alpha and beta are folders inside this project and they represent groups in this project. A group is a container in this repo and what it represents is really not relevant for this question. The repo X also has files in its root level; project.conf is an example of such a file.

I have a class called Project that abstracts projects such as X. The Project class has a method load() that builds an in-memory representation.

class Project(object):

    def load(self):
        for entry in os.listdir(self.root):
            path = os.path.join(self.root, entry)
            if os.path.isdir(path):
                group = Group(path)
                self.groups.append(group)
                group.load()
            else:
                # process files ...

To unit test the load() method by mocking the file system, I have:

import unittest
from unittest import mock
import Project

class TestRepo(unittest.TestCase):

    def test_load_project(self):
        project = Project("X")

        with mock.patch('os.listdir') as mocked_listdir:
            mocked_listdir.return_value = ['alpha', 'beta', 'project.conf']
            project.load()
            self.assertEqual(len(project.groups), 2)

This does mock os.listdir successfully. But I can't trick Python to treat mocked_listdir.return_value as consisting of files and directories.

How do I mock either os.listdir or os.path.isdir, in the same test, such that the test will see alpha and beta as directories and project.conf as a file?

Answer 1

You could use pyfakefs (source, docs), a very handy lib to test operations on a fake file system.

if you use pytest, it has a plugin, all file system functions already got patched, you just need to use its fs fixture:

import os

def test_foo(fs):
    fs.CreateFile('/home/x/alpha/1')
    fs.CreateFile('/home/x/beta/2')
    fs.CreateFile('/home/x/p.conf')    
    assert os.listdir('/home/x/')) == ['alpha', 'beta', 'p.conf']

or if you prefer unittest:

import os
import unittest

from pyfakefs import fake_filesystem_unittest
   
class TestRepo(fake_filesystem_unittest.TestCase):

    def setUp(self):
        self.setUpPyfakefs()

    def test_foo(self):
        os.makedirs('/home/x/alpha')
        os.makedirs('/home/x/beta')
        with open('/home/x/p.conf', 'w') as f:
            f.write('foo')
        self.assertEqual(os.listdir('/home/x/'), ['alpha', 'beta', 'p.conf'])


if __name__ == "__main__":
    unittest.main()

Answer 2

It will depend, of course, on exactly which os functions you use, but it looks like mock.patch.multiple on os is just what you need. (Note that you may not need to patch path; many of its functions are lexical-only and do not care about the actual filesytem.)