'AWS Glue Dynamic Frame Pushdown Predicate List
When using pushdown predicate with AWS Glue Dynamic frame, how does it iterate through a list?
For example, the following list was created to be used as a pushdown predicate:
day=list(p_day.select('day').toPandas()['day'])
month=list(p_month.select('month').na.drop().toPandas()['month'])
year=list(p_year.select('year').toPandas()['year'])
predicate = "day in (%s) and month in (%s) and year in (%s)"%(",".join(map(lambda s: "'"+str(s)+"'",dat))
,",".join(map(lambda s: "'"+str(s)+"'",month))
,",".join(map(lambda s: "'"+str(s)+"'",year)))
Let's say it returns this:
"day in ('07','15') and month in ('11','09','08') and year in ('2021')"
How would the push down predicate read this combination/list?
Is it:
| day | month | year |
|---|---|---|
| 07 | 11 | 2021 |
| 15 | 11 | 2021 |
| 07 | 09 | 2021 |
| 15 | 09 | 2021 |
| 07 | 08 | 2021 |
| 15 | 08 | 2021 |
-OR-
| day | month | year |
|---|---|---|
| 07 | 11 | 2021 |
| 15 | 11 | 2021 |
| 15 | 08 | 2021 |
| 15 | 09 | 2021 |
I have a feeling that this list is read like the first table rather than the latter... But, it's the latter that I would like to pass through as a pushdown predicate. Does creating a list essentially cause a permutation? It's as if the true day, month, and year combination is lost in the list which should be 11/7/2021, 11/15/2021, 08/15/2021, and 09/15/2021.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
