'How to convert a 4 dimensional numpy array to a pyspark dataframe?
I have a 4-dimensional NumPy array and a 3-dimensional one that look like this:
data = numpy.zeros((1985, 5, 512, 512), dtype=float)
labels = numpy.ones((1985, 5, 1), dtype=float)
These arrays are currently a small portion of the data I'll be using, so I decided to convert them into PySpark dataframes so that I can handle the large datasets. However, all the information I found shows how to convert 2-dimensional arrays into PySpark dataframes. Does it work with more dimensions? If so, how? Thanks in advance.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|