ExternalSecret referencing to AWS parameter store, updated value there only take effect after I delete my service pod, why?

Question

I have my backend service (my-service) which uses environment variables from AWS parameter store.

my-service manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-service
spec:
  ...
  ...
  spec:
    containers:
    - name: my-service
      image: my_service_image:latest
      ...
      envFrom:
        - secretRef:
              name: my-param-store

As you can see above, the envFrom section specified a secret reference named my-param-store, which is a AWS Parameter Store. Its manifest looks like this:

apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: my-param-store
spec:
  backendType: systemManager
  data:
    - key: /my-service/MY_SECRET
      name: MY_SECRET

After I deployed all these. My service is running fine. Then I want to see that if I change the value stored in AWS parameter store, my-service should not be able to handle request anymore. To verify that I did the following:

Step 1. I changed the value of the key /my-service/MY_SECRET in AWS parameter store. I expect my service would fail to handle request, but it still handle requests successfully. I was thinking "Hmm... probably I should manually delete that my-param-store object & create a new one so that the new value in AWS parameter store would start take effect.

Step 2. So, I kubectl delete externalsecret my-param-store. Then I deployed again, the new external secret object is created in cluster.

Step 3. Then, I tried again sending request to my-service, I expect this time it should fail to handle request, but NO, my-service still can handle request successfully.

Step 4. Finally, I decided to delete my-service pod (kubectl delete pod my-service-58fb12). After did that, k8s spin up a new my-service pod. And this time my-service indeed failed at handling request due to the value in AWS parameter store was changed.

My question is, Why should I delete my-service pod in order to have the new value in AWS parameter store starts taking effect? I mean why it doesn't take effect after I deleted my-param-store & created a new one(in step 2)? What is the mechanism of k8s behind that makes only deleting my-service pod would have the new value of key in AWS parameter store take effect in my case?

Answer 1

... uses environment variables from AWS parameter store.

envFrom:

• secretRef: name: my-param-store

As you can see above, the envFrom section specified a secret reference named my-param-store

Environment variable created from secret resource will not be updated when the secret changed. See the official note here.