Testing if a pandas DataFrame exists

Question

In my code, I have several variables which can either contain a pandas DataFrame or nothing at all. Let's say I want to test and see if a certain DataFrame has been created yet or not. My first thought would be to test for it like this:

if df1:
    # do something

However, that code fails in this way:

ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

Fair enough. Ideally, I would like to have a presence test that works for either a DataFrame or Python None.

Here is one way this can work:

if not isinstance(df1, type(None)):
    # do something

However, testing for type is really slow.

t = timeit.Timer('if None: pass')
t.timeit()
# approximately 0.04
t = timeit.Timer('if isinstance(x, type(None)): pass', setup='x=None')
t.timeit()
# approximately 0.4

Ouch. Along with being slow, testing for NoneType isn't very flexible, either.

A different solution would be to initialize df1 as an empty DataFrame, so that the type would be the same in both the null and non-null cases. I could then just test using len(), or any(), or something like that. Making an empty DataFrame seems kind of silly and wasteful, though.

Another solution would be to have an indicator variable: df1_exists, which is set to False until df1 is created. Then, instead of testing df1, I would be testing df1_exists. But this doesn't seem all that elegant, either.

Is there a better, more Pythonic way of handling this issue? Am I missing something, or is this just an awkward side effect all the awesome things about pandas?

Answer 1

In my code, I have several variables which can either contain a pandas DataFrame or nothing at all

The Pythonic way of indicating "nothing" is via None, and for checking "not nothing" via

if df1 is not None:
    ...

I am not sure how critical time is here, but since you measured things:

In [82]: t = timeit.Timer('if x is not None: pass', setup='x=None')

In [83]: t.timeit()
Out[83]: 0.022536039352416992

In [84]: t = timeit.Timer('if isinstance(x, type(None)): pass', setup='x=None')

In [85]: t.timeit()
Out[85]: 0.11571192741394043

So checking that something is not None, is also faster than the isinstance alternative.

Answer 2

If the dataframe is stored as a dictionary value, you could test for its existence this way:

import pandas as pd

d = dict()
df = pd.DataFrame()

d['df'] = df

## the 'None' is default but including it for the example
if d.get('df', None) is not None:
    ## get df shape
    print(df.shape)
else:
    print('no df here')