'Look after certain value in BigQuery

I am using an append-only BigQuery table and using high watermark logic to attempt to speed up the query logic. My query looks as below.

    query = f"""
    SELECT thing.*, 
    thing2.*
    FROM {table1_id}
    FULL OUTER JOIN {table2_id}
    ON thing.hash = thing2.hash
    WHERE address = "{address}" AND thing.highwater_mark_field > {highwatermark} AND thing2.highwater_mark_field > {highwatermark};
    """

I am using a high watermark to attempt to allow BigQuery to scan the table only where the values of highwater_mark_field (int) are greater than the highwatermark variable.

Unfortunately, BigQuery does not by default look after the value as it does not know it is incremental. I was wondering if there is a way to force it to ONLY look AFTER the high watermark in the table to speed up the query as the table has billions of rows.

I am tempted to try and brute force this by grabbing the row ID and trying to iterate from there but I cannot find information if this would work, I was wondering if anyone can tell me if this would work or has a more elegant solution?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source