Can a kubernetes pod respond to a request when it is removed from a service endpoint?

Question

I know that the moment the pod receives a deletion request, it is deleted from the service endpoint and no longer receives the request. However, I'm not sure if the pod can return a response to a request it received just before it was deleted from the service endpoint. If the pod IP is missing from the service's endpoint, can it still respond to requests?

Answer 1

1. As long as the pod is deleted and container stopped, it cannot respond to requests, no matter whether it is removed from the service's endpoints.
2. If the pod container is still alive, it can respond to requests, no matter whether you visit it through svc or not.

Answer 2

There are many reasons why Kubernetes might terminate a healthy container (for example, node drain, termination due to lack of resources on the node, rolling update).

Once Kubernetes has decided to terminate a Pod, a series of events takes place:

1 - Pod is set to the “Terminating” State and removed from the endpoints list of all Services

At this point, the pod stops getting new traffic. Containers running in the pod will not be affected.

2 - preStop Hook is executed

The preStop Hook is a special command or http request that is sent to the containers in the pod. If your application doesn’t gracefully shut down when receiving a SIGTERM you can use this hook to trigger a graceful shutdown. Most programs gracefully shut down when receiving a SIGTERM, but if you are using third-party code or are managing a system you don’t have control over, the preStop hook is a great way to trigger a graceful shutdown without modifying the application.

3 - SIGTERM signal is sent to the pod

At this point, Kubernetes will send a SIGTERM signal to the containers in the pod. This signal lets the containers know that they are going to be shut down soon. Your code should listen for this event and start shutting down cleanly at this point. This may include stopping any long-lived connections (like a database connection or WebSocket stream), saving the current state, or anything like that. Even if you are using the preStop hook, it is important that you test what happens to your application if you send it a SIGTERM signal, so you are not surprised in production!

4 - Kubernetes waits for a grace period

At this point, Kubernetes waits for a specified time called the termination grace period. By default, this is 30 seconds. It’s important to note that this happens in parallel to the preStop hook and the SIGTERM signal. Kubernetes does not wait for the preStop hook to finish. If your app finishes shutting down and exits before the terminationGracePeriod is done, Kubernetes moves to the next step immediately. If your pod usually takes longer than 30 seconds to shut down, make sure you increase the grace period. You can do that by setting the terminationGracePeriodSeconds option in the Pod YAML.

5 - SIGKILL signal is sent to pod, and the pod is removed

If the containers are still running after the grace period, they are sent the SIGKILL signal and forcibly removed. At this point, all Kubernetes objects are cleaned up as well.

I hope this gives a good idea of the Kubernetes termination lifecycle and how to handle a Pod termination gracefully.

Based on this article.