'Text is not interpreting the null in source dataframe(csv) as blank/null in the resultant dataframe(txt.)
I read the source data from a CSV file to a dataframe. Some columns have null values as seen in dataframe A.
Now, dataframe B is the resultant data frame while writing the data in text. As you see, the text format gives an error. The null values are ignored and have been merged in a single column which is correct.
However, dataframe C is what I am expecting. Are there any ways through which I could achieve dataframe C?
Dataframes A, B, C are below:
sample data
A)
+---+---+---+----+----+----+----+--------+
|_c0|_c1|_c2| _c3| _c4| _c5| _c6| _c7|
+---+---+---+----+----+----+----+--------+
|abc| 1| 2|null|null|null|null|10/12/22|
+---+---+---+----+----+----+----+--------+
B)
+----------------+
| data|
+----------------+
|abc,1,2,10/12/22|
+----------------+
However, this is what I am expecting.
C)
+--------------------+
| data |
+--------------------+
|abc,1,,,,,2,10/12/22|
+--------------------+
code snippet
file_location = "/FileStore/tables/sample_data.csv"
file_type = "csv"
infer_schema = "false"
first_row_is_header = "false"
delimiter = ","
df = spark.read.format(file_type)
.option("inferSchema", infer_schema)
.option("header", first_row_is_header)
.option("sep", delimiter)
.load(file_location)
df.show() # dataframe A
from pyspark.sql.functions import concat_ws
single_column_df = df.select(concat_ws(",", *df.columns).alias('data'))
destination = "/FileStore/tables/output10.txt"
file_format = "text"
single_column_df.write.format(file_format).save(destination)
single_column_df.show() # dataframe B
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
