'Insufficient 'DISKS_TOTAL_GB' quota on Dataproc Serverless
I am ingesting 200 plus files into bigquery on dataproc serverless. The input files are not huge at all. All of them are in few mbs. Still many jobs are failing with error "Insufficient 'DISKS_TOTAL_GB' quota"
When I checked disk, i have 150TB plus space before job started. It was dataproc job which ate all space. Is there any way we can configure persistent disk that get allocated to each dataproc job>
Solution 1:[1]
You can set the disk sizes for the jobs as mentioned here
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Jayadeep Jayaraman |