'Great Expectations: How to add a partition (column partition) in an Athena External Table in a checkpoint reference in GE?
The setup is GE v3 and I am using AWS Athena as a Data Source. However, I couldn't find a way to tell the "expectation" that the table in actually partitioned with a relative path in S3 like 's3://path_to_bucket/path_to_db/path_to_table/version=2022-02-01'.
I am looking for a way to run a expect_column_values_to_be_unique but when it takes the next parition 's3://path_to_bucket/path_to_db/path_to_table/version=2022-02-02', it returns a fail.
However, it only happens because it takes both 'version=2022-02-02' and 'version=2022-02-01' to verify the uniqueness on the table's primary key.
My question is: how can I pass version as an argument to the expectation / checkpoint?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|