'Creating a new column in PySpark, and indexing a row in the dataframe that returns a value 1 before
I have a pyspark dataframe that I am cleaning and it is based off of logic previously built into Alteryx.
Inside the dataframe, you picture there being the following columns: col1 = 'Part', col2 = 'Tool'
I am creating a new column called 'Fill' and this column is created with this alteryx logic assigned to it:
'''if !IsNull(Part) then Part ELSE [row-1:Fill]'''
How would in Pyspark would I go about taking the previous row and putting its 'Fill' value in the new row if the condition is met?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
