'Running Dask in preconfigured AWS auto scaling instances
We want to split a task and work on them in distributed way as volume of data is very high. The work can be achieved only via specialized EC2 instances. Question is, is it possible to create dask cluster with specialized EC2 instances as workers and non specialized instance as dask scheduler? I do not see any API in dask.distributed to provide pre-configured dask worker cluster. The EC2Cluster API creates scheduler and worker. But we would like to create EC2 auto scaling instances for workers and separate machine for scheduler. Can someone help on achieving or guiding on the way to do it.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|