'Look after certain value in BigQuery
I am using an append-only BigQuery table and using high watermark logic to attempt to speed up the query logic. My query looks as below.
query = f"""
SELECT thing.*,
thing2.*
FROM {table1_id}
FULL OUTER JOIN {table2_id}
ON thing.hash = thing2.hash
WHERE address = "{address}" AND thing.highwater_mark_field > {highwatermark} AND thing2.highwater_mark_field > {highwatermark};
"""
I am using a high watermark to attempt to allow BigQuery to scan the table only where the values of highwater_mark_field (int) are greater than the highwatermark variable.
Unfortunately, BigQuery does not by default look after the value as it does not know it is incremental. I was wondering if there is a way to force it to ONLY look AFTER the high watermark in the table to speed up the query as the table has billions of rows.
I am tempted to try and brute force this by grabbing the row ID and trying to iterate from there but I cannot find information if this would work, I was wondering if anyone can tell me if this would work or has a more elegant solution?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
