'AWS Glue, multiple bookmarks within same job?
Lets say I have a script that is loading multiple frames with different schemas
job.init()
DefinitionDyf = glueContext.create_dynamic_frame_from_options(connection_type = "s3", connection_options = {"paths": [DefinitionPath], "recurse": True}, format = "csv",
format_options= {'withHeader': True}, transformation_ctx="DefinitionBookmark")
TypeDyf = glueContext.create_dynamic_frame_from_options(connection_type = "s3", connection_options = {"paths": [TypePath], "recurse": True}, format = "csv",
format_options= {'withHeader': True}, transformation_ctx="TypeBookmark")
I do some transformations and then I wrote to another bucket and I end the script with
job.commit()
Would both bookmarks be updated or just the first one? Is it recommended to split up bookmarks like this? most of the examples I saw only had one bookmark for each job.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
