'"Can not merge type error" when converting Pandas dataframe to Spark dataframe
panda_df:
| Column | DataType |
|---|---|
| OrganizationID | Int64 |
| Name | object |
| OrgCode | Int64 |
| StartDate | datetime64[ns] |
| EndDate | datetime64[ns] |
| ApplicationID | Int64 |
panda_df have ,NaT and nan values.
spark_df=spark.createDataFrame(panda_df)
when converting to the Spark dataframe getting -
TypeError: field ApplicationID: Can not merge type <class 'pyspark.sql.types.StructType'> and <class 'pyspark.sql.types.LongType'>
How to handle the ,NaT and nan values and get the Int data type instead of LongType.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|