Connection reset - Docker swarm

Question

We are deploying several services in docker swarm (16 services in total) in a single master node. Most of these services are developed in Quarkus, some of them compiled in native mode, others are java compiled because of their dependencies.

When the services are in use everything works fine, but if they are waiting to be used for more than 15 minutes we start to receive this message:

Caused by: java.net.SocketException: Connection reset,
    at java.net.SocketInputStream.read(SocketInputStream.java:186),
    at java.net.SocketInputStream.read(SocketInputStream.java:140),
    at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137),
    at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153),
    at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280),
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138),
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56),
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259),
    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163),
    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157),
    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273),
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125),
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272),
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186),
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89),
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110),
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185),
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83),
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56),
    at org.jboss.resteasy.client.jaxrs.engines.ManualClosingApacheHttpClient43Engine.invoke(ManualClosingApacheHttpClient43Engine.java:302),

It doesn't matter if it is native compilation or Java compilation, in all of them the novelty is reproduced.

This message is presented when trying to consume the API also built in Quarkus, its consumption is through Swarm's internal network using the name of the service to resolve its location.

To consume the api we build a jar, in which we implement the services interfaces as indicated in the guide https://quarkus.io/guides/rest-client using additionally org.eclipse.microprofile.faulttolerance for the retry.

Answer 1

Because of lack of communication through the socket your connection is lost. I found few solutions here: Apache HttpClient throws java.net.SocketException: Connection reset if I use it as singletone