What is the downside of increasing the maximum number of Pods per node in AKS with CNI?

Question

When creating a new node pool in Azure Kubernetes Services, the default maximum number of Pods per Node differs between CNI and Kubenet (and the tool used to create the node pool). According to the docs, the default setting is usually 30 with CNI, but 110 with Kubenet.

Why does CNI have a lower default setting and what are the downsides of increasing the actual value, e.g., to 110, like with Kubenet?

Answer 1

110 is the default size definet by Kubernetes upstream. With Kubnet only the nodes get an IP on your Azure subnet. All the pods gets IPs from a logical network on the nodes and they are using NAT (iptables) to communicate with the Azure Network.

When you use Azure CNI, Azure pre-allocates IPs in you subnet. You can read this here:

Each node is configured with a primary IP address. By default, 30 additional IP addresses are pre-configured by Azure CNI that are assigned to pods scheduled on the node.

This means for every node with max pods 30 you would need 31 addresses free in your subnet.If your subnet is to small, you could not add any new node bcs Azure needs those 31 IP addresses to add this node.

I think max pods 30 is more like a security value bcs people tend to use /24 subnets. With max pods 110 you could only add 2 nodes to this subnet. If your AKS is running you can not change the subnet or the max pods size - this would require a new AKS to be created.

There is no downside on using max pods 110 on your nodes except that you need to size your subnet accordingly and you would need more planning. We are mostly using /16 vnets with /21 subnets with max pods 110 on our AKS clusters:

Clusters configured with Azure CNI networking require additional planning. The size of your virtual network and its subnet must accommodate the number of pods you plan to run and the number of nodes for the cluster.