'Write PySpark dataframe to BigQuery "Numeric" datatype
For simplicity, I've a table in BigQuery with one field of type "Numeric". When I try to write a PySpark dataframe, with one column, to BigQuery it keeps on raising the NullPointerException. I tried converting pyspark column into int, float, string, and even encode it but it keeps on throwing the NullPointerException. Even after spending 5 to 6 hours, I'm unable to figure it out myself or on the internet that what is the issue here and what should be the exact pyspark dataframe column type for mapping it to BigQuery Numeric column type. Any help or direction would be of great help. Thanks in advance.
Solution 1:[1]
For anyone who faces the same issue, you just have to cast the column to decimal type.
from pyspark.sql.types import DecimalType
subscriber_df_deu.withColumn('column', col('column').cast(DecimalType()))
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Malina Dale |