How to use REST API as source for Lookup activity in Azure Data Factory

Question

I am trying to incrementally load data from a ServiceNow data source into an Azure SQL table as per the guide from Microsoft https://docs.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-portal. This uses 2 lookup activities to find 2 dates, I then filter the source data in the copy activity to only return and insert rows where the sys_updated_on date is between the 2 lookup values.

I would now like to lookup a value from a REST API dataset. However, I do not get the option to choose my REST dataset in the lookup activity. It just does not appear as an option. The REST URL is setup to return me one date value which I need to pass into the WHERE clause of my source in the copy data. If I cannot retrieve this value in the lookup, how else can I pass it to my WHERE clause?

Currently I use activity('LookupOldWaterMarkActivity').output.firstRow.watermarkvalue and convertTimeZone(activity('LookupNewWaterMarkActivity').output.firstRow.watermarkvalue

Thanks

Answer 1

As per the Microsoft official document, the Rest dataset is supported in lookup activity.

You can post feedback from an Azure Data factory or raise a support request for fixing the issue.

As a workaround, you can create an HTTP dataset with JSON format and use the output value in later activities.