'is it possible for command time on Databricks to increase to almost double without changing the cluster specifications? what could be causing that?

to store data in s3 bucket form Databricks, I used to write the following:

df.write.format("delta").mode("overwrite").save("s3://.....")

the code above used to take 3.27 minutes, it takes now 7.37 minutes, using the same cluster configuration and the same data.

performancepysparkcluster-computingdatabricks


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source

Related Questions