'Export data from Dynamodb incrementally (say: of last 24 hrs)
I want to put a batch process to export upserts data from Dynamodb of last 24 hours only (every night). It's going to run in midnight and fine with some latency.
I know these options:
#1) scan the table(and having a updateDateTime attribute). But its going to be costly($$) as data grows.
#2) create a GSI on UpdateDate, and query this index using this partition key(hot partition issues, as well as extra cost of having a GSI and RCUs).
Tried this also https://aws.amazon.com/blogs/aws/new-export-amazon-dynamodb-table-data-to-data-lake-amazon-s3/ - but it exports all the data till a datetime (and not last x hours)
Tried Athena and AWS Glue option, but exported data is not incremental
Please advise if there is a better way?
Note: I dont need all the events happening on DynamoDB (otherwise streams is a option). I want to export the end of the day snapshot(but incremental)
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
