'Prevent Delta table merge to create a lot of small files?
I am merging an update dataframe into a big Delta table. When doing so I noticed that the amount of parquet files increases drastically. I know that afterwards I can perform a vacuum command on that table with a retention period of 0 hours. However, I think this is pretty inefficient. Is there a way to turn off the delta versioning? I thought of converting that delta table into a parquet table but then I'm not able to perform a merge anymore...
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
