Redis pool failing to get connection

Question

We are experiencing issues with our Redis after some hours on production. Services start to randomly throw this errors:

Caused by: redis.clients.jedis.exceptions.JedisException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:51)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:271)
... 19 common frames omitted
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:449)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
    at redis.clients.util.Pool.getResource(Pool.java:49)
    ... 22 common frames omitted

We are using GCP Redis with a Jedis Pool. After a restart of the service it returns to normal operation. Production settings allow a lot of connections:

jedis:
  pool:
    max-active: 4000
    max-idle: 4000
    min-idle: 1000

We have added monitoring in order to understand the numbers when the errors occur. :

log.info("Redis pool stats. active {}, idle {}, waiters {}", jedisPool.getNumActive(), jedisPool.getNumIdle(), jedisPool.getNumWaiters());

Redis pool stats. active -1, idle -1, waiters -1

What does the -1 mean?

All the places where Jedis connections are used, are with try-with-resources or use the SpringBoot RedisTemplate. We are quite sure that we dont leave open connections.

Has anyone experienced something like that before and can give us a clue where to search?

Answer 1

What does the -1 mean?

It means the pool is either "not initialized" or "closed".