'Dataproc: Can user create workers of different instance types?
scenario: master: x1 machine type workers: x2-machine type, x3-machine type.
For the above scenario: AWS EMR instance fleet allows users to create different worker instance types. From dataproc console, I noticed options is for only, N-worker types(all worker types of same type). Does Dataproc HDI also support this option of creating different instance types for worker?
Solution 1:[1]
Dataproc supports 2 types of worker group: (primary) workers and secondary workers. But they must have the same machine type as of Q2 2022, there is only one flag --worker-machine-type for it.
Note that it does allow secondary workers to have different disk type, disk size, accelerator, etc. See this doc for more details.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Dagang |