k8s deployment imagePullPolicy set to ifNotPresent still pulls images

Question

Hello all I am trying to figure out what is making our GCP k8s pull images even with the image has been set as a specific version and the policy is ifnotpresent. Is there a way to check the image age inside the cluster? Anyone has any idea what might cause this. By documentation if the policy is notpresent and the image doesn't have tag latest it should pull it only once. What happened is that one of the images that we use, the image repo went down for some reason which caused imagepullbackoff, the cluster is running fine for the past year and a half, suddenly the repo drops and we got a problem. I know that one of the solutions is to use our own image registry but still, wondering why is this happening.

Answer 1

With current information it is hard to indicate any particular reason.

Depends if you are using Google Kubernetes Engine or Google Compute Engine with Kubeadm. If you have configured Artifact Registry, Container Registry or you are downloading images from Dockerhub

In general, you are right about the IfNotPresent policy. In Understanding Kubernetes Image Pull Policies blog, you can find default behavior:

You don’t have to specify an image pull policy. When a Pod lacks a policy, Kubernetes will infer your intentions from the image’s tag. If you’ve supplied a specific tag (such as my-image:my-release), the image will only be pulled if the tag doesn’t already exist on the Kubelet node. This policy is called IfNotPresent.

Similar information can be found in the Openshift Image pull policy overview:

IfNotPresent - Only pull the image if it does not already exist on the node.

In short, if you have some autoscaler which creates a node and image is huge, it will need to download it to that node.

You have also mention:

What happened is that one of the images that we use, the image repo went down for some reason which caused imagepullbackoff

This behavior might be related with the way how the image was pushed, like in this guide

Useful links:

• Pulling cached Docker Hub images
• Preparing a Google Kubernetes Engine environment for production
• Deploying to Google Kubernetes Engine

To sum up

It's hard to determine the exact reason why GCP behaved like this with current information. Logs, and more details of behavior and env configuration would be required. Only things which comes to my mind is that there was a new node which required to download this new image or reuploading this image to Artifact/Container registry changed some background values (some kind of ID).