'pyspark 2.4 How to Format nested field to datetime from epoch
After reading in Parquet Files using pyspark 2.4 I would like to format a field within a struct and array. I would like to replace the doctime field that is in epoch time(e.g. 1280620800000) to a formatted datetime. How would I construct a withcolumn and how to format epoch to datetime?
root
|-- Key: struct
| |-- subkey: array
| | |-- subkey1: struct
| | | |-- name: string
| | | |-- val: struct
| | | | |-- text: string
|-- Data: struct
| |-- subdata: array
| | |-- subdata1: struct
| | | |-- key1: long
| | | |-- value: struct
| | | | |-- numeric: double
| | | | |-- string: string
| | | |-- doctime: long
| | | |-- tag: integer
| | | |-- rptime: long
Any help is greatly appreciated.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
