'Error while trying to convert a Spark dataframe into Pandas dataframe with ".toPandas()" function (Method arrowEnabled([]) does not exist)
I use pyspark (version = 2.4.5) and pyarrrow (version = 0.14.1). I tried to convert a spark dataframe into a Pandas dataframe as follows:
df_pandas = df.toPandas(),
where df represents the spark dataframe and where df_spark represents the Pandas dataframe.
When I try to convert a spark dataframe (composed by 4 string-type columns) to a pandas datafrace with the ".toPandas()" function, I get the following error:
py4j.protocol.Py4JError: An error occurred while calling o42.arrowEnabled. Trace: py4j.Py4JException: Method arrowEnabled([]) does not exist
I have also checked the compatibility between the versions of pyspark and pyarrow, and the versions are compatible with each other.
How can the above-mentioned error be fixed?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|