'Do you know any way how I can overwrite certain partitions in a folder, and leave other partitions unchanged?
do you know any way how I can overwrite certain partitions in a folder, and leave other partitions unchanged?
Solution 1:[1]
If your dataframe contains only data from the partitions you would like to overwrite then you need to set spark.sql.sources.partitionOverwriteMode to dynamic (Take a look here as well).
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Guy |