iPython debugger raises `NameError: name ... is not defined`

Question

I can't understand the following exception that is raised in this Python debugger session:

(Pdb) p [move for move in move_values if move[0] == max_value]
*** NameError: name 'max_value' is not defined
(Pdb) [move for move in move_values]
[(0.5, (0, 0)), (0.5, (0, 1)), (0.5, (0, 2)), (0.5, (1, 0)), (0.5, (1, 1)), (0.5, (1, 2)), (0.5, (2, 0)), (0.5, (2, 1)), (0.5, (2, 2))]
(Pdb) max_value
0.5
(Pdb) (0.5, (0, 2))[0] == max_value
True
(Pdb) [move for move in move_values if move[0] == 0.5]
[(0.5, (0, 0)), (0.5, (0, 1)), (0.5, (0, 2)), (0.5, (1, 0)), (0.5, (1, 1)), (0.5, (1, 2)), (0.5, (2, 0)), (0.5, (2, 1)), (0.5, (2, 2))]
(Pdb) [move for move in move_values if move[0] == max_value]
*** NameError: name 'max_value' is not defined

Why is it sometimes telling me max_value is not defined and other times not?

Incidentally, this is the code immediately prior to the debugger starting:

max_value = max(move_values)[0]
best_moves = [move for move in move_values if move[0] == max_value]
import pdb; pdb.set_trace()

I am using Python 3.6 running in PyCharm.

AMENDED UPDATE:

After more testing it appears that local variables are not visible within list comprehensions within a pdb session when I do the following from an iPython REPL or in PyCharm:

$ ipython
Python 3.6.5 | packaged by conda-forge | (default, Apr  6 2018, 13:44:09) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pdb; pdb.set_trace()
--Call--
> /Users/billtubbs/anaconda/envs/py36/lib/python3.6/site-packages/IPython/core/displayhook.py(247)__call__()
-> def __call__(self, result=None):
(Pdb) x = 1; [x for i in range(3)]
*** NameError: name 'x' is not defined

But in a regular Python REPL it works:

$ python
Python 3.6.5 | packaged by conda-forge | (default, Apr  6 2018, 13:44:09) 
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pdb; pdb.set_trace()
--Return--
> <stdin>(1)<module>()->None
(Pdb) x = 1; [x for i in range(3)]
[1, 1, 1]

I tested above with versions 3.4, 3.5, 3.6 so it does not appear to be version dependent.

UPDATE 2

Please note, the above test ('AMENDED UPDATE') is problematic because it uses import pdb; pdb.set_trace() in the interactive REPL.

Also, the original problem is not limited to iPython.

See answer by user2357112 below for a comprehensive explanation of what is going on here.

Sorry if I caused any confusion!

Answer 1

A possible solution/workaround is to run

 globals().update(locals())

before running the list comprehension in (i)pdb.

Answer 2

Inspired by Kobi T's answer, if you want the workaround to apply without manual intervention every time this problem comes up, you can subclass TerminalPdb (ipython's debugger, but it should also work by subclassing Pdb). The key is extending TerminalPdb's (really, Pdb's, as it's inherited) default method. The default method is the one called whenever you run a line of code from Pdb's debugger prompt.

As user2357112-supports-monica's answer described well, this function uses exec to run the code, and typically captures any errors raised and prints them out instead of raising them. You can make use of that fact that for the particular problem of list comprehensions (and closures more generally) that appears here, a NameError gets raised. Specifically, if that's the raised error you can intercede to apply Kobi T's workaround (a version of globals().update(locals())), and then exec the code again--if the NameError was raised because of the closure problem described here, the code should now run with no errors; if not, the NameError will get raised again and the output will be as expected for that error.

An added benefit here relates to how globals are handled. For the purposes of the code run by exec, the global/local variables it sees can be explicitly defined as inputs to exec. As a result, instead of updating the workspace's globals(), you can update just the copy that gets sent to exec, keeping your workspace a little less affected when you want to run these list comprehensions.

import sys
import traceback
from IPython.terminal.debugger import TerminalPdb

class LcTerminalPdb(TerminalPdb):
    # the default method is what runs code written to the prompt of the debugger
    def default(self, line):
        # most of this method directly copies the original one, but there's no
        # good way to add the NameError handling separately from the original
        # code
        if line[:1] == '!': line = line[1:]
        locals = self.curframe_locals
        globals = self.curframe.f_globals
        try:
            code = compile(line + '\n', '<stdin>', 'single')
            save_stdout = sys.stdout
            save_stdin = sys.stdin
            save_displayhook = sys.displayhook
            try:
                sys.stdin = self.stdin
                sys.stdout = self.stdout
                sys.displayhook = self.displayhook
                exec(code, globals, locals)
            ''' BELOW IS THE CODE ADDED TO Pdb's default()'''
            except NameError:
                # NameError occurs when a list comprehension requires variables
                # to be bound in its closure, but isn't able to because of how
                # exec handles local variables; putting the variable in the
                # global dictionary works, and this code takes the sledgehammer
                # approach of assigning *all* locals to globals, so we don't
                # have to be picky about which variable exactly was needed
                try:
                    tempGlobal = globals.copy()
                    tempGlobal.update(locals)
                    exec(code, tempGlobal, locals)
                except:
                    raise
            '''ABOVE IS THE CODE ADDED TO Pdb's default()'''
            finally:
                sys.stdout = save_stdout
                sys.stdin = save_stdin
                sys.displayhook = save_displayhook
        except:
            self._error_exc()

    # TerminalPdb doesn't directly call _error_exc, which was originally defined
    # in Pdb, so we can't import it from there, and it's underscored, so it
    # isn't immediately accessible from TerminalPdb's parent Pdb. However, it's
    # a simple function so I'm just replicating it here.
    def _error_exc(self):
        exc_info = sys.exc_info()[:2]
        self.error(traceback.format_exception_only(*exc_info)[-1].strip())

Here's the result we want:

>> import pdb; pdb.set_trace()
(Pdb) x=5;[x for i in range(3)]
*** NameError: name 'x' is not defined
(Pdb) q
>> import LcTerminalPdb; LcTerminalPdb.set_trace()
ipdb> x=5;[x for i in range(3)]
[5, 5, 5] 

This solution will work if you call LcTerminalPdb.set_trace(). Alternatively, you can save this as a module on the Python path, in that module define a set_trace method that calls LcTerminalPdb.set_trace() (below) and then set the PYTHONBREAKPOINT environment variable to point to it.

[appended to file with class code from above]
def set_trace(frame=None):
    """
    Start debugging from `frame`.
    If frame is not specified, debugging starts from caller's frame.
    """
    LcTerminalPdb().set_trace(frame or sys._getframe().f_back)

I made a quick LcTerminalPdb git repo with this file and instructions on how to get that part working for anyone interested.