'Query DynamicFrame in AWS Glue

i have a question about DynamicFrames, i want to run query in the moment when i get data from the catalog and avoid get all the data and then query. At the moment it's not a lot of data, but maybe in the future. Now, i testing with the parameter "push_down_predicate", but only run when the data is partitioned.

Example:

df_data = glueContext.create_dynamic_frame.from_catalog(database = "datalake-raw",
                                                            table_name = "data",
                                                            transformation_ctx = "df_data",
                                                            push_down_predicate = "( CampaignKey == '1')")

Know another way to run that? is this possible to do?

Thanks!!



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source