kubernetes scheduler customisation

Question

I'd like to change the scheduler behavior to schedule pods only base on the number of pods. We have a umbrella helm chart that includes 40-50 subcharts. Each of our developer has his own testing environment. In order to save cost, I removed the resource request for each pod so that I can run many pods per node. This worked well for months until recently we on-boarded a few more developers and we are now running 1500+ pods. Sometimes some nodes become unresponsive because two many pods running on it and it get overloaded. I noticed that even I added new nodes, pods didn't get to scheduled to new nodes. Part of the reason I guess is because I removed cpu & mem request. Can I change the scheduler configuration to make it only schedule base on the number of pods?

Answer 1

For changing the scheduler behavior, basically Kubernetes ships with a default scheduler that is described here. “If the default scheduler does not suit your needs you can implement your own scheduler”. Moreover, you can even run multiple schedulers simultaneously alongside the default scheduler and instruct Kubernetes what scheduler to use for each of your pods.

Also you need to have a kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts.

To determine if a scheduler is responsible for scheduling a specific Pod, the spec.schedulername field in a PodTemplate or Pod manifest must match the scheduler Name field of the kubeSchefulerProflie. All Schedulers running in the cluster must have unique names.

For more information on scheduler customization please refer to Doc.