How does gRPC connection work on kubernetes service ClusterIP

Question

I'm probably missing something in my understanding of networking and gRPC. I want to start with an example to explain how I understand the networking.

I have a deployment with 3 replicas named app in default namespace they have pod IP:

10.3.0.1, 10.3.0.2, 10.3.0.3

I have a ClusterIP service for the deployment called app-service, it has an IP of:

10.24.0.0

The kube-dns will have a record that maps the app-service.default.svc.cluster.local -> 10.24.0.0. The kube-proxy on every node will see the configuration and update the netfilter with mapping 10.24.0.0 -> 10.3.0.1, 10.3.0.2, 10.3.0.3.

Now in my cluster I have another client pod that makes a gRPC call to app-service.default.svc.cluster.local.

What I expect to happen is app-service.default.svc.cluster.local will resolve to a single IP 10.24.0.0, and gRPC will create a single sub channel and try to establish a long-live connection.

This call will get out of the pod and to the node and go pass the netfilter at which point 10.24.0.0 becomes 10.3.0.1 and finally arrives on the first pod.

Now there is a second call from the client, it goes through the netfilter at which point 10.24.0.0 becomes 10.3.0.2, this call ends up on another pod which had no idea they had a connection established?

Also I see these blogs that mention gRPC will create sticky session to a single pod IP, but I thought the pod IP will not be resolved inside the application/grpc client but in the netfilter.

Answer 1

While your thinking process might sound correct, it works differently.

When you have a single clusterIP service it has a single A DNS record. So connection will look like this:

So at the given moment there will be only one/single persistent connection, other two pods won't be used. kube-proxy has this mapping inside using netfilter, but it's not a proper physical loadbalancer, no PODs' IPs will be exposed and client will be aware only about service's clusterIP. If a new connection is created, already created connection will be used with a help of grpc multiplexing.

What you need here is a headless service type. What it does is:

For headless Services that define selectors, the endpoints controller creates Endpoints records in the API, and modifies the DNS configuration to return A records (IP addresses) that point directly to the Pods backing the Service.

And schema will look differently now:

In other words, with headless service type service's FQDN will be resolved in 3 PODs IP addresses where gRPC client can establish different connections.

For more details please read a good article (where I took these pictures from) - Balancing gRPC Traffic in K8S Without a Service Mesh.

Also service mesh can be used for loadbalancing (proxy loadbalancing),