'AWS glue Workers vs Concurrency
I am looking for suggestions on how to set glue job parameters. Currently I get "Concurrent runs exceeded"as exception.
What can i do to scale and run the job properly
Solution 1:[1]
Concurrent runs exceeded means that you tried to run the job while an instance of the job is already running and you set the MaxConcurrentRuns property of the job. You can scale a job not only by running it concurrently, which I also would advise to, since this most of the Databases / Filesystems you are reading and writing to and from are probably not ACID compliant, you can also increase the NumberOfWorkers to deploy additional executor nodes.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Robert Kossendey |