DockerHub: sha digest doesn't match

Question

I want to determine the sha digest for the latest docker image in DockerHub:

if I try to pull the latest image I can see the digest

# docker pull mysql:latest
...
Digest: sha256:c93ba1bafd65888947f5cd8bd45deb7b996885ec2a16c574c530c389335e9169
Status: Downloaded newer image for mysql:latest
docker.io/library/mysql:latest

so the digest is c93ba1

But if I go to https://hub.docker.com I'll see

So there are 2 digests: 511ca265b41c and 9a355d5c4ec0.

Also if I try to pull with any of the specified digests:

# docker pull mysql@sha256:9a355d5c4ec0351a954b11b494c597cd4e6ab2c8a04ce69c5f8332819890c43e
Error response from daemon: manifest for mysql@sha256:9a355d5c4ec0351a954b11b494c597cd4e6ab2c8a04ce69c5f8332819890c43e not found: manifest unknown: manifest unknown

# docker pull mysql@sha256:511ca265b41cabb694fda409b8ae87fb0a83db15cfb8429d581c33c7aafacddf
Error response from daemon: manifest for mysql@sha256:511ca265b41cabb694fda409b8ae87fb0a83db15cfb8429d581c33c7aafacddf not found: manifest unknown: manifest unknown

But if I try with the one that I got from the pull command it works:

docker pull mysql@sha256:c93ba1bafd65888947f5cd8bd45deb7b996885ec2a16c574c530c389335e9169
sha256:c93ba1bafd65888947f5cd8bd45deb7b996885ec2a16c574c530c389335e9169: Pulling from library/mysql
Digest: sha256:c93ba1bafd65888947f5cd8bd45deb7b996885ec2a16c574c530c389335e9169
Status: Image is up to date for mysql@sha256:c93ba1bafd65888947f5cd8bd45deb7b996885ec2a16c574c530c389335e9169
docker.io/library/mysql@sha256:c93ba1bafd65888947f5cd8bd45deb7b996885ec2a16c574c530c389335e9169

What 'm I doing wrong? And is there a way to get the digest without pulling the image?

Update: Repeated the same procedure again on MacOs using Docker Desktop 2.2

docker pull mysql:8
8: Pulling from library/mysql
...
Digest: sha256:f91e704ffa9f19b9a267d9321550a0772a1b64902226d739d3527fd6edbe3dfe
Status: Downloaded newer image for mysql:8
docker.io/library/mysql:8

Then on docker hub

Then

docker image inspect mysql:8
[
    {
        "Id": "sha256:c8ad2be69a220e93826a6308458627b8d5624dc981050fabf950e5de5a7a08a8",
        "RepoTags": [
            "mysql:8"
        ],
        "RepoDigests": [
            "mysql@sha256:f91e704ffa9f19b9a267d9321550a0772a1b64902226d739d3527fd6edbe3dfe"
        ]

And

docker pull mysql@sha256:a592539c5a616b6642bb48822688b6917b373a1293638f9268e8da33e5e9dd1c
sha256:a592539c5a616b6642bb48822688b6917b373a1293638f9268e8da33e5e9dd1c: Pulling from library/mysql
Digest: sha256:a592539c5a616b6642bb48822688b6917b373a1293638f9268e8da33e5e9dd1c
Status: Downloaded newer image for mysql@sha256:a592539c5a616b6642bb48822688b6917b373a1293638f9268e8da33e5e9dd1c
docker.io/library/mysql@sha256:a592539c5a616b6642bb48822688b6917b373a1293638f9268e8da33e5e9dd1c

docker pull mysql@sha256:f91e704ffa9f19b9a267d9321550a0772a1b64902226d739d3527fd6edbe3dfe
sha256:f91e704ffa9f19b9a267d9321550a0772a1b64902226d739d3527fd6edbe3dfe: Pulling from library/mysql
Digest: sha256:f91e704ffa9f19b9a267d9321550a0772a1b64902226d739d3527fd6edbe3dfe
Status: Image is up to date for mysql@sha256:f91e704ffa9f19b9a267d9321550a0772a1b64902226d739d3527fd6edbe3dfe
docker.io/library/mysql@sha256:f91e704ffa9f19b9a267d9321550a0772a1b64902226d739d3527fd6edbe3dfe

And when I do the same on Linux VM:

sudo docker pull mysql:8
8: Pulling from library/mysql
...
Digest: sha256:f91e704ffa9f19b9a267d9321550a0772a1b64902226d739d3527fd6edbe3dfe
Status: Downloaded newer image for mysql:8
docker.io/library/mysql:8

So I have no idea what is happening here

Answer 1

What you see when pulling the image is the digest for the manifest list or OCI index when the image is packaged as a multi-platform image (even when that packaging only has a single platform). To use the example from the question:

$ regctl manifest get mysql@sha256:f91e704ffa9f19b9a267d9321550a0772a1b64902226d739d3527fd6edbe3dfe --format body | jq .
{
  "manifests": [
    {
      "digest": "sha256:a592539c5a616b6642bb48822688b6917b373a1293638f9268e8da33e5e9dd1c",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      },
      "size": 2828
    }
  ],
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "schemaVersion": 2
}

That is a docker manifest list that includes a single platform specific manifest (for linux/amd64) with digest sha256:a592539c5a616b6642bb48822688b6917b373a1293638f9268e8da33e5e9dd1c that you see reported on Docker Hub. Docker tracks the manifest list on a pull, but it always dereferences a multi-platform image to a single platform when pulling to the docker engine.

You typically want to use the multi-platform digest when pinning a digest in a deployment. That allows the same digest can be used on different platforms, though in this specific case that's less important. To get that digest, tools like regclient, crane, and skopeo can be used. And buildx also includes a hidden command docker buildx imagetools inspect that can do this.

Answer 2

This is not a feature, and it has an open long-running Github issue here, where you can chip in and upvote it to be fixed quicker.

In a similar question on SO you can also find a workaround script which uses curl and will get you correct server-side digests: https://stackoverflow.com/a/64309017/9962007

Otherwise the CLI is still out of sync with the Hub, even docker inspect. For instance:

Docker Hub (link):

bitnami/minideb:buster
Digest:sha256:b4d0417f742c591cad881ba5458719edb2b8166103413ffc4e1e480455892097

vs. local docker inspect:

IMAGE_NAME_TAGGED=bitnami/minideb:buster && docker inspect $IMAGE_NAME_TAGGED | grep sha
        "Id": "sha256:0def18b69c0b60f4d4ceb22f5aad596dffbb6d1b6089aff15158a4cde5276e2d",
            "bitnami/minideb@sha256:0a96a8ec61c52b638c824bc3463c9009844d80d5e2b1dfe77b6c10a31a27684d"
                "sha256:4c16ec6258b6cd4630c3d8e7c1389981064e2719d560d5f08b9cfff1082fa86f"

I'm not sure if switching to podman will improve situation, as in this reopened issue users describe similar problems with digests mismatches.

Answer 3

Curiously, when I visit dockerhub, mysql:latest (now) has a different digest that it reports as updated one month ago:

https://hub.docker.com/layers/mysql/library/mysql/latest/images/sha256-a65e1689b806ccb757887565a3c1d8e7467f14621012d472076cad4117eb06f3

In your example, c93va1 is the correct hash for your architecture. Are you using Mac? The other hashes, if valid are for the images for for Linuxes on be ARM and AMD64. Could that explain the difference your see?

I find dockerhub to be less than ideal when dealing with manifests. Google Container Registry shows the manifest of images along with the hash and so it's easier to corroborate.

See this answer to a question that I had that's related and hopefully helps: https://stackoverflow.com/a/57878742/609290