'Why is pandas.grouby.mean so much faster than paralleled implementation

I was using the pandas grouby mean function like the following on a very large dataset:

import pandas as pd
df=pd.read_csv("large_dataset.csv")
df.groupby(['variable']).mean() 

It looks like the function is not using multi-processing, and therefore, I implemented a paralleled version:

import pandas as pd 
from multiprocessing import Pool, cpu_count 

def meanFunc(tmp_name, df_input): 
    df_res=df_input.mean().to_frame().transpose()
    return df_res 

def applyParallel(dfGrouped, func):
    num_process=int(cpu_count())
    with Pool(num_process) as p: 
        ret_list=p.starmap(func, [[name, group] for name, group in dfGrouped])
    return pd.concat(ret_list)

applyParallel(df.groupby(['variable']), meanFunc)

However, it seems that pandas implementation is still way faster than my parallel implementation.

I am looking at the source code for pandas groupby, and I see that it is using cython. Is that the reason?

def _cython_agg_general(self, how, alt=None, numeric_only=True,
                        min_count=-1):
    output = {}
    for name, obj in self._iterate_slices():
        is_numeric = is_numeric_dtype(obj.dtype)
        if numeric_only and not is_numeric:
            continue

        try:
            result, names = self.grouper.aggregate(obj.values, how,
                                                   min_count=min_count)
        except AssertionError as e:
            raise GroupByError(str(e))
        output[name] = self._try_cast(result, obj)

    if len(output) == 0:
        raise DataError('No numeric types to aggregate')

    return self._wrap_aggregated_output(output, names)


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source