'Custom container label in Dataflow UI is impossible to resolve to its identity
When I run Dataflow jobs with a custom container the UI shows a nice and friendly
sdk_container_image gcr.io/project/image:latest
Clearly at the time of execution the label is resolved to the real container identity (sha checksum). Unfortunately the value is not surfaced in the UI or REST request. Nor is it visible in the pipeline or worker logs. This means that if labels are moved (which happens all the time with "latest") it is impossible to figure out the version of the container used by the given run.
While there's a way to work around this deficiency by always resolving the label locally before starting jobs, it's quite onerous. I know the internal tools at Google include proper resolved labels in the UI. I'd appreciate the same level of polish in GCP offerings.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|