'Pyspark schema treeString to schema object
I have the output of the printSchema() (a treeString) in a txt file. Is there a pyspark API to parse the string and create the schema object?
Example Input:
root
|-- num: long (nullable = true)
|-- letter: string (nullable = true)
Example Output: spark schema object
StructType(List(StructField(num,LongType,true),StructField(letter,StringType,true)))
Solution 1:[1]
As for as I know, there is no such thing to convert a Big output print (came from printSchema() that basically returns None. to an actual pyspark object.
In order to do it, you will have to create the schema on your own.
Write a parser that does it for you, but its pretty useless, just work with DF.schema that return's what's you need.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 |