Kubernetes - increase kubectl log size

Question

I have an app running in a kubernetes managed docker container, using Azure Kubernetes Service (AKS). I can output logs to a text file for the app deployment using:

kubectl logs deployment/my-deployment > out.txt

This gives me a file of around max 28Mb. When I get the docker logs for the same container on the physical VM using docker logs ..., the log file is much bigger (up to 120Mb+).

How can I increase the size of the available kubectl logs for the deployment? If this option is available, then it would likely be an option that increases the size of the available kubectl logs for the pod that holds the container, as the pod and deployment logs are identical.

It's not the docker configuration in /etc/docker/daemon.json that's limiting the kubectl logs, as that's set to 50Mb. I've read that it's the underlying docker configuration that kubernetes uses, but that doesn't seem to be the case, as mykubectl logs are being truncated to around 28Mb.

Answer 1

Kubectl logs might read logs with a default log rotation, meaning the logrotate service is active.

Check the content of cat /etc/logrotate.d/docker-containers, as in this issue, for confirmation.

As explained in unofficial-kubernetes:

An important consideration in node-level logging is implementing log rotation, so that logs don't consume all available storage on the node. Kubernetes uses the logrotate tool to implement log rotation.

Kubernetes performs log rotation daily, or if the log file grows beyond 10MB in size.
Each rotation belongs to a single container; if the container repeatedly fails or the pod is evicted, all previous rotations for the container are lost.
By default, Kubernetes keeps up to five logging rotations per container.