'Is there a good way for surfacing the time data was last updated in a Workshop Module?
Is there a good way to surface when a dataset backing an object was last built in a Workshop module? This would be helpful for giving users of the module a view on data freshness.
Solution 1:[1]
The ideal situation is that your data encodes the relevant information about how fresh it is; for instance if your object type represent "flights" then you can write a Function that sorts and returns the most recent flight and present it's departure timestamp as the "latest" update, since it represents the most recent data available.
The next best approach would be to have a last_updated column or similar that's either coming from the source system or added during the sync step. If the data connection is a JDBC or similar connection, this would be straightforward; something like select *, now() as last_updated_timestamp. If you have a file-based connection, you might need to get a bit more creative. This still falls short of accurately conveying the actual "latest data" available in the object type, but at least let's the user know when the last extract from the source system occured.
There are API endpoints for various services in Foundry related to executing schedules and builds, but metadata from these can be misleading if presented to users as an indication of data freshness because they don't actually "know" anything about the data itself - for example you might get the timestamp of when the latest pipeline build started, but if the source system has a 4 hour lag before data is synced to the relevant export tables, then you'll still be "off". So again, best to get this from inside your data wherever possible.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Logan Rhyne |