Spring boot, log failing health check

Question

We use spring-boot(2.5.6) in kubernetes

Many of the dependencies we use include healthchecks for instance RedisHealthIndicator,CouchbaseHealthIndicator etc

When one of these health checks fails and the overall application health fails, the pod is restarted by kubernetes.

However there is no indication why it failed, spring does not log health check failures, instead relying on the Healthcheck itself to log a message. Which is not the case for the built in health checks.

So from the outside it appears that kubernetes has killed this pod for 'no reason' and we have to assume it was the health check

Does spring have a 'health check change event' so that I can log which bean has failed?

Or otherwise track the 'down' state of the health on an individual basis

https://github.com/spring-projects/spring-boot/issues/22632 This issue is similar but they explicitly state they will not log failures

Answer 1

I've fought with this awhile myself. I'm not sure why they've taken that stance on logging health failures, but what is worse is the current implementation is incredibly unfriendly to try and inject that kind of functionality into.

In the end, the work around I settled on involved wrapping the health contributors so that I can log messages if they report not-up. The wrapper itself is pretty simple:

public class LoggingHealthIndicator implements HealthIndicator {
    private static final Logger LOG = LoggerFactory.getLogger(LoggingHealthIndicator.class);
    private final String name;
    private final HealthIndicator delegate;

    public LoggingHealthIndicator(final String name, final HealthIndicator delegate) {
        this.name = name;
        this.delegate = delegate;
    }

    @Override
    public Health health() {
        final Health health = delegate.health();

        if (!Status.UP.equals(health.getStatus())) {
            if (health.getDetails() == null || health.getDetails().isEmpty()) {
                LOG.error("Health check '{}' {}", name, health.getStatus());
            }
            else {
                LOG.error("Health check '{}' {}: {}", name, health.getStatus(), health.getDetails());
            }
        }

        return health;
    }
}

You could of course do whatever you want; Raise an application event, further tweak when and what you log, etc. As fancy as you like.

As far as making it actually used, that's where it gets a little annoying. It involves replacing the HealthContributorRegistry with our own enhanced version.

    /**
     * Replicated from {@link org.springframework.boot.actuate.autoconfigure.health.HealthEndpointConfiguration}.
     *
     * Note that we lose the {@link org.springframework.boot.actuate.autoconfigure.health.HealthEndpointConfiguration.AdaptedReactiveHealthContributors},
     * since it is private. Technically its private to the package-scoped class it's a child of, so we lose twice really.
     */
    @Bean
    @SuppressWarnings("JavadocReference")
    public HealthContributorRegistry healthContributorRegistry(final Map<String, HealthContributor> contributors, final HealthEndpointGroups groups) {
        return new LoggingHealthContributorRegistry(contributors, groups.getNames());
    }    

public class LoggingHealthContributorRegistry extends AutoConfiguredHealthContributorRegistryCopy {

    private static HealthContributor loggingContributor(final Entry<String, HealthContributor> entry) {
        return loggingContributor(entry.getKey(), entry.getValue());
    }

    private static HealthContributor loggingContributor(final String name, final HealthContributor contributor) {
        if (contributor instanceof HealthIndicator){
            return new LoggingHealthIndicator(name, (HealthIndicator)contributor);
        }
        return contributor;
    }

    public LoggingHealthContributorRegistry(Map<String, HealthContributor> contributors, Collection<String> groupNames) {
        // The constructor does not use `registerContributor` on the input map entries
        super(contributors.entrySet().stream().collect(Collectors.toMap(Entry::getKey, LoggingHealthContributorRegistry::loggingContributor)),
              groupNames);
    }

    @Override
    public void registerContributor(String name, HealthContributor contributor) {
        super.registerContributor(name, loggingContributor(name, contributor));
    }
}

A note about AutoConfiguredHealthContributorRegistryCopy: it's literally just a copy of the AutoConfiguredHealthContributorRegistry class that happens to be package-scoped and so isn't inheritable (unless you don't mind playing package games)