How to do incremental load for Appflow on Demand

Question

I am updating record from s3 to salesforce using Appflow on demand.What property i can define in the CFT to make my appflow on demand incremental based on the s3 last modified timestamp.

Answer 1

There is no way to perform incremental data load for on-demand app flows.

Ideally, this use-case might be fulfilled by “Run Flow on Event” settings under “Flow Trigger” option under Appflow, but this option is not available for S3 Bucket as a Source.

I found some steps that should address this requirement or similar requirement of reading only the latest file/data from the S3 bucket.

My understanding is that you would like to configure the Appflow which runs every time when there is an S3 Bucket Change Event. (when a new file is added to the S3 bucket)

However, as a workaround you can use the Lambda function to run the flow with “S3 Event notification” as a Trigger to the lambda.

The Architecture will be as below.

S3 Bucket (with new events) > Lambda > Appflow

Create a Lambda python program that runs the “flow” every time there is a new event in the S3 Bucket

Answer 2

You are mixing columns and rows, maybe you should rename the variable row to record so you see better what is happening. Unfortunately, I have to guess as to how the data file could look like...

The dict variable (try not to use this name, this is actually a built-in function and you don't want to overwrite it) is creating two columns, "artist", and "name", which seem to have values like [1.2]. So, dict (try to print it) could look like {"artist":[2.0], "name":[3.1]}, which is a single row, two column entity.

artist     name
2.0        3.1

Try to get into pandas, use the df = pd.read_csv() and df[df.something > 0.3] style notation to filter tables, using the csv package is better suited for truly tricky data wrangling.