'Create a BigQuery Table everyday from a CSV file stored in Cloud Storage

On GCP I have a new csv added everyday at 2 am in a Cloud Storage folder (format is something like mydata_yyyymmdd.csv).

I am trying to schedule an upload of this file at 2:30 am everyday in a Big Query Table.

I succeeded in creating a dataflow that constantly screen my Cloud Storage folder and constantly update my BigQuery table if a new file is added but I don't find this optimal as :

my dataflow is running all day long while I only need to run it once a day (increasing costs)
this solution doesn't create a new Big Query table everyday, it just appends all csv to my Big Query table

Can you help me with the tools I should use in GCP to achieve this ?

Thanks a lot for your help, it is very much appreciated

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source

'Create a BigQuery Table everyday from a CSV file stored in Cloud Storage

Sources

Related Questions