Jobs not getting cleaned up until 30+ minutes after TTL has passed

Question

We are using the Job ttlSecondsAfterFinished attribute to automatically clean up finished jobs. When we had a very small number of jobs (10-50), the jobs (and their pods) would get cleaned up approximately 60 seconds after completion. However, now that we have ~5000 jobs running on our cluster, it takes 30 + minutes for a Job object to get cleaned after completion.

This is a problem because although the Jobs are just sitting there, not consuming resources, we do use a ResourceQuota (selector count/jobs.batch) to control our workload, and those completed jobs are taking up space in the ResourceQuota.

I know that jobs only get marked for deletion once the TTL has passed, and are not guaranteed to be deleted immediately then, but 30 minutes is a very long time. What could be causing this long delay?

Our logs indicate that our k8s API servers are not under heavy load, and that API response times are reasonable.

Answer 1

Solution 1

How do you use the Job ttlSecondsAfterFinished? You can specify .spec.ttlSecondsAfterFinished to the value what you need. Below is the example from official documentation

apiVersion: batch/v1
kind: Job
metadata:
  name: pi-with-ttl
spec:
  ttlSecondsAfterFinished: 100
  template:
    spec:
      containers:
      - name: pi
        image: perl
        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: Never

And please note this:

Note that the TTL period, e.g. .spec.ttlSecondsAfterFinished field of Jobs, can be modified after the job is created or has finished. However, once the Job becomes eligible to be deleted (when the TTL has expired), the system won't guarantee that the Jobs will be kept, even if an update to extend the TTL returns a successful API response.

For more information: https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/#updating-ttl-seconds

Solution 2

As it mentioned above in the comment field, you can try to play with kube-controller-manager and increase the number of TTL-after-finished controller workers that are allowed to sync concurrently by using the following flag option:

kube-controller-manager --concurrent-ttl-after-finished-syncs int32 Default: 5