TcpClient.ConnectAsync throws SocketException Connection refused on Linux

Question

I have a TcpListener listening on localhost. Next application creates TcpClient and connects to the server.

Both server application and client application are written in .net core 3.1 and published as self contained apps.

Everything works fine on Windows.

On Linux (Debian inside a Docker image, latest Debian image) the TcpClient throws SocketException. Both applications are in the same Docker image and in the same folder.

System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (111): Connection refused 127.0.0.1:5005 at System.Net.Sockets.Socket.BeginConnectEx(EndPoint remoteEP, Boolean flowContext, AsyncCallback callback, Object state) at System.Net.Sockets.Socket.UnsafeBeginConnect(EndPoint remoteEP, AsyncCallback callback, Object state, Boolean flowContext) at System.Net.Sockets.Socket.BeginConnect(EndPoint remoteEP, AsyncCallback callback, Object state) at System.Net.Sockets.Socket.BeginConnect(IPAddress address, Int32 port, AsyncCallback requestCallback, Object state) at System.Net.Sockets.Socket.BeginConnect(String host, Int32 port, AsyncCallback requestCallback, Object state) at System.Net.Sockets.TcpClient.BeginConnect(String host, Int32 port, AsyncCallback requestCallback, Object state) at System.Net.Sockets.TcpClient.<>c.b__28_0(String targetHost, Int32 targetPort, AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory1.FromAsyncImpl[TArg1,TArg2](Func5 beginMethod, Func2 endFunction, Action1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions) at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func5 beginMethod, Action1 endMethod, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions) at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func5 beginMethod, Action1 endMethod, TArg1 arg1, TArg2 arg2, Object state) at System.Net.Sockets.TcpClient.ConnectAsync(String host, Int32 port)

I can confirm that TcpListener has started successfully (logging in app).

Server code (logging omitted):

var listener = new TcpListener(IPAddress.Loopback, 5005);
listener.Start();
var client = await listener.AcceptTcpClientAsync();
using (var networkStream = client.GetStream())
{
  ...
}

Client code:

var client = new TcpClient(AddressFamily.InterNetwork);
// also tried "localhost" instead of ip address
var connectTask = client.ConnectAsync("127.0.0.1", 5005);
var isConnectTaskFinished = await Task.WhenAny(connectTask, Task.Delay(timeout, m_cancellationTokenSource.Token)) == connectTask;
...
client.Close();

I'm not sure what can be wrong and how to find the root cause. Do I need to allow the port on Debian or set some permissions for both apps?