Memory explosion on TCP server during debug on client

Question

I hope it is the right place to ask. I use a nuget, WatsonTcp for both tcp client and server. The settings to the keep alive is this:

Server.Keepalive.EnableTcpKeepAlives = true;
Server.Keepalive.TcpKeepAliveInterval = 2;   
Server.Keepalive.TcpKeepAliveTime = 2;       
Server.Keepalive.TcpKeepAliveRetryCount = 2;

And the send information for client is this:

                IEnumerator<string> Clients = connectedIpAddresses.GetEnumerator();

                // if for some reason the server had a ClientDisconnected event
                // happened during the data preparation we should not trye to send 
                if (Clients != null)
                {
                    while (Clients.MoveNext())
                    {
                        // get current connected client IP
                        string IpPort = Clients.Current;

                        if (IpPort != null)
                        {
                            try
                            {
                                if (!disconnectedClients.Contains(IpPort))
                                {
                                    Server.SendAsync(IpPort, dataToCients);
                                }
                            }
                            catch (Exception ex)
                            {
                                string error = "From inside CollectDataAndSendToClient: Problem sending to client " + ex.Message;
                                AddTextToDebugInfoLog(error);
                                ZeissLogger.Logger.Error(error);
                            }
                        }
                        else
                        {
                            // in case the server called 
                            if (!connectedIpAddresses.Contains(IpPort))
                            {
                                string error = "From inside CollectDataAndSendToClient: Problem sending to client at " + IpPort;

                                AddTextToDebugInfoLog(error);
                            }
                        }


                        // if we have more than one client 
                        if (connectedIpAddresses.Count > 1)
                        {
                            Thread.Sleep(15);
                        }
                    }
                }

Basicly it works pretty stable. But we notice a thing: If a client is on debug at the same point for a long time,(Not on the part that handle the tcp client, but in other part of the software) the memory usage of the server starts to rise, and once we tried and it reached to ceash because of memory leak. We put this code to handle it:

        while (Running)
        {
            // call garbage collector to free un used memory.
            //
            // Without this call we get memory leaks due to TCP cable disconnect
            // and/or debugging application that blocks.
            //
            GC.Collect();

            Thread.Sleep(1000);
            if (GC.GetTotalMemory(true) > (SERVER_MAX_MEMORY_BEFORE_CLEANOUT))                {
                ZeissLogger.Logger.Error("Server memory leak detected. Disconnecting all clients !!");
                AddTextToDebugInfoLog("Server memory leak detected. Disconnecting all clients !!");

                // add a small delay to let the log item added 
                Thread.Sleep(500);

                // during this we protect agains changes in the list
                lock(lockForIpList)
                {
                    foreach(string ip in connectedIpAddresses)
                    {
                        Server.DisconnectClient(ip);

                        // add a small delay in between
                        Thread.Sleep(15);
                    }

                    // clear the lists before continuing
                    connectedIpAddresses.Clear();
                    disconnectedClients.Clear();

                    // add a large delay in order to give the GC time to clear the used memory
                    Thread.Sleep(10000);
                    
                    Server.Start();
                }                }

            // add a delay after every iteration in order to not cause stress
            Thread.Sleep(2000);

But the disconnection of the debugger client doesn't work until we not continue on the client side. We afraid that in some points the serer will crash if lets say the application of the client will be stuck. It t like the debug at same point without forward put the client on a middle state where it is not connected and not disconnected. Any Ideas? Is this problem will happen only on debug, and there is nothing to do?(I can as a solution restart the server, and than we saw that the debug client don't reconnect, although he have a task that does it every few seconds, but we prefer not to do so).