'Memcached: why memcached server constantly drops connection?

I have two spring apps, in both I use spymemcached. And memcached client is created like this

private MemcachedClient buildClient() {
    String address = this.serverAddress;
    MemcachedClient result;
    if (StringUtils.isNotEmpty(address)) {
      log.info("connecting to memcached server {}", address);
 
      try {
        result = new MemcachedClient(
            new ConnectionFactoryBuilder()
                .setTranscoder(new CustomSerializingTranscoder())
                .setProtocol(protocol)
                .setTimeoutExceptionThreshold(2)
                .build(),
            AddrUtil.getAddresses(address));
      } catch (IOException e) {
        throw new IllegalStateException("Can't create client", e);
      }
 
      log.info("connected to memcached server {}", address);
    } else {
      log.warn("cannot initialize memcached client: memcached address is not set");

      result = null;
    }
    return result;
  }

One spring app put (always replace one) key:value in memcached. Another one reads this key. And the app that reads value constantly loose connection.

Memcached config:

[r.gomboev@netris-gps-stage60-01 netris]$ sudo systemctl status memcached
● memcached.service - Memcached
   Loaded: loaded (/usr/lib/systemd/system/memcached.service; enabled; vendor preset: disabled)
   Active: active (running) since Пн 2022-01-24 18:29:39 MSK; 1 day 1h ago
 Main PID: 16747 (memcached)
    Tasks: 6
   Memory: 18.9M
   CGroup: /system.slice/memcached.service
           └─16747 /usr/bin/memcached -u memcached -p 11211 -m 64 -c 1024 -vv

But a problem is memcached server constantly drops connections.

memcached logs:

Jan 25 13:21:50 memcached-server-01 memcached: <33 new auto-negotiating client connection
Jan 25 13:22:44 memcached-server-01 memcached: <33 connection closed.
Jan 25 13:23:19 memcached-server-01 memcached: <33 new auto-negotiating client connection
Jan 25 13:23:33 memcached-server-01 memcached: <33 connection closed.
Jan 25 13:23:55 memcached-server-01 memcached: <33 new auto-negotiating client connection
Jan 25 13:26:34 memcached-server-01 memcached: <33 connection closed.
Jan 25 13:31:36 memcached-server-01 memcached: <33 new auto-negotiating client connection
Jan 25 13:45:03 memcached-server-01 memcached: <33 connection closed.
Jan 25 16:11:43 memcached-server-01 memcached: <33 new auto-negotiating client connection
Jan 25 18:45:12 memcached-server-01 memcached: <33 connection closed.
Jan 25 19:39:36 memcached-server-01 memcached: <33 new auto-negotiating client connection
Jan 25 19:41:11 memcached-server-01 memcached: <33 connection closed.

when I look at connection through lsof

on app server 81.166
[user@test-dmz-02 ~]$ sudo lsof -i -P -n | grep 11211 | grep 82.80
java      12150      cctv   90u  IPv6 1037210363      0t0  TCP 10.200.81.166:35742->10.200.82.80:11211 (ESTABLISHED)
connection established 

but on memcached server 82.80
[user@memcached-server-01]$ sudo lsof -i -P -n | grep 11211 | grep 81.166
no connection

memcached has next stats

Escape character is '^]'.
stats
STAT pid 16747
STAT uptime 93258
STAT time 1643131435
STAT version 1.4.15
STAT libevent 2.0.21-stable
STAT pointer_size 64
STAT rusage_user 10.729242
STAT rusage_system 17.056611
STAT curr_connections 15
STAT total_connections 62
STAT connection_structures 20
STAT reserved_fds 20
STAT cmd_get 738
STAT cmd_set 84624
STAT cmd_flush 0
STAT cmd_touch 0
STAT get_hits 737
STAT get_misses 1
STAT delete_misses 0
STAT delete_hits 0
STAT incr_misses 0
STAT incr_hits 0
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT touch_hits 0
STAT touch_misses 0
STAT auth_cmds 0
STAT auth_errors 0
STAT bytes_read 156369146
STAT bytes_written 100524068
STAT limit_maxbytes 67108864
STAT accepting_conns 1
STAT listen_disabled_num 0
STAT threads 4
STAT conn_yields 0
STAT hash_power_level 16
STAT hash_bytes 524288
STAT hash_is_expanding 0
STAT bytes 85
STAT curr_items 1
STAT total_items 84623
STAT expired_unfetched 0
STAT evicted_unfetched 0
STAT evictions 0
STAT reclaimed 0
END

I can't find the trigger Why connection is closed by memcached server. Can anybody show me the way to find out the solution? Thank you



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source