'Incremental pipeline with parameter using Data Fusion
Is it possible to create a parameter from a table in Big Query, in this case the max ID (last row inserted), to use as start point for an incremental pipeline using Data Fusion?
I'm using Data Fusion for the pipeline, it pulls data from a MySQL database and inserts in a Big Query table, but I want to pull incremental data from MySQL, using the currently max ID from the Big Query table as start point, that means to pull from MySQL only the rows that have ID greater than the max ID in Big Query.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|