AWS Quicksight: How to Get the latest Data from S3

Question

My Quicksight currently takes everything in the S3 bucket

(S3 sample) https://i.stack.imgur.com/cO8kL.png

But S3 keep changing folder base on the date so 01/,02/,03/ and so on Is there a way to only take the latest data not all of it?

[this my current manifest:]

{
"fileLocations": [
    {
        "URIPrefixes": [
            "https://sample-S3bucket.amazonaws.com/"
        ]}
],
"globalUploadSettings": {
    "format": "JSON" 
}

}

There might be a simple solution that I might not know about.

Answer 1

You could set Quicksight to read from another bucket and set a lambda that is triggered when a a new file is uploaded into your existing bucket. This lambda would:

1. Remove any files from the bucket which QuickSight is reading from
2. Copy over the latest file into the bucket
3. Creates a QuickSight SPICE ingestion via api.