'Best way to schedule a NodeJS process on GCP that takes more than 10 minutes
I have a nightly batch load job (14.7 million records)into a mySQL table, we have been using the GCP cron, but it occasionally times out. Do you have any suggestions or best practice tips for this type of scheduled process? We are using the following sql statements:
- delete from
- Load data local infile
Our time is greatly reduced if we use a truncate statement rather than a delete, however with truncate we cannot rollback if there is a problem with the load.
Solution 1:[1]
2 parts in your question:
- firstly, cloud scheduler can handle task up to 30 minutes. Therefore, if you use Cloud Run to run your script, you are no longer limited to 9 minutes as you are with Cloud Functions (if my assumption is the right one)
- Secondly, I can recommend, if it's possible in your database, to start by loading the data into a temporary table. if the load is OK, truncate your table and insert select the data from the temporary table to the target table.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | guillaume blaquiere |