'AWS Glue - glueContext.purge_table leads to "No such file or directory 's3://abc..."

I'm using AWS Glue, and I want to overwrite a Glue catalog with a Glue job. During my Glue job, I call

glueContext.purge_table(glue_database, glue_table, options={"retentionPeriod": 0})

My next line is me trying to write out the current dataframe out to the catalog:

sink = glueContext.write_dynamic_frame_from_catalog( frame=master_dyf, database=glue_database, table_name=glue_table, additional_options=additionalOptions)

But this throws an error of:

An error occurred while calling o362.pyWriteDynamicFrame. No such file or directory 's3://dev.some.bucket/dev/somepath/part-00009-d324e8e6-dbd5-41e2-b216-c1933d0c120a-c000.snappy.parquet'

What's going on here? I can't figure out why purging a catalog is also removing a my parquets.

Any help is greatly appreciated.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source