"FutureWarning: Dropping of nuisance columns in DataFrame reductions" warning when using df.mean()

Question

I have a dataframe that looks something like this:

   col1   col2 col3
0     1   True  abc
1     2  False  def
2     3   True  ghi

When I run df.mean(), it shows a warning:

>>> df.mean()
<stdin>:1: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError.  Select only valid columns before calling the reduction.
col1    2.000000
col2    0.666667
dtype: float64

How do I solve this warning?

Answer 1

Numeric functions such as mean, median, sem, skew, etc., only support dealing with numeric values. If you look at the data types of your columns...

>>> df.dtypes
col1     int64
col2      bool
col3    object
dtype: object

...you can see that the dtype of col1 is int64, which mean can handle, because it's numeric. Likewise, the dtype of col2 is bool, which Python, pandas, and numpy essentially treat as ints, so mean treats col2 as if it only contains 1 (for True) and 0 for False.

The dtype of col3, however, is object, the default dtype for strings, which is basically a generic type to encapsulate any type of data that pandas can't understand. Since it's not numeric, mean has no idea how to deal with it. (After all, how would you compute the mean of abc and def?)

There are a few ways to solve this problem, but "ignoring it" isn't one of them, because, as the warning indicates, in a future version of pandas, this warning will become an error that will stop your code from running.

1. Use numeric_only=True. This will cause mean to skip columns that aren't numeric — col3 in this case:

   >>> df.mean(numeric_only=True)
   col1    2.000000
   col2    0.666667
   dtype: float64
   

   (Notice how col3 is omitted).

2. Select only the columns you need to operate on:

   >>> df[['col1', 'col2']].mean()
   col1    2.000000
   col2    0.666667
   dtype: float64