How can I manage the memory leak on HoloLens 2?

Question

I have a client-server app. HoloLens 2 is my client and communicates with the server using UDP. My MRTK profiler shows me the app consumes memory incrementally and after 2-3 mins app crashes. I send some packets from the server to HoloLens. I tried to call GC but it doesn't work either. I tried to remove using blocks and threads but the result didn't change. I use only one port. Should I connect, get packets and dispose of the socket for each packet? (this is so terrible solution) Anyway, any idea how can I fix the leak?

private void Socket_MessageReceived(Windows.Networking.Sockets.DatagramSocket sender,
Windows.Networking.Sockets.DatagramSocketMessageReceivedEventArgs args)
{
    //Read the message that was received from the UDP  client.
    Stream streamIn = args.GetDataStream().AsStreamForRead();
    MemoryStream ms = Task.Run(() => ToMemoryStream(streamIn)).Result;
    byte[] msgData = ms.ToArray();
    ThreadManager.ExecuteOnMainThread(() =>
    {
        Task.Run(() => HandleData(msgData));
     });
}

private async Task<MemoryStream> ToMemoryStream(Stream input)
{
    try
    {
        using (MemoryStream ms = new MemoryStream())
        {
            // Read and write in blocks of 4K.
            byte[] block = new byte[0x1000];
            while (true)
            {
                int bytesRead = input.Read(block, 0, block.Length);
                if (bytesRead == 0)
                {
                    return ms;
                }
                ms.Write(block, 0, bytesRead);
            }
        }
    }
    finally { }
}

private async System.Threading.Tasks.Task HandleData(byte[] data)
{
    using (Packet packet = new Packet(data))
    {
        int packetLength = packet.ReadInt();
        data = packet.ReadBytes(packetLength);
    }
        
    ThreadManager.ExecuteOnMainThread(() =>
    {
        using (Packet packet = new Packet(data))
        {
            int packetId = packet.ReadInt();
            // Call appropriate method to handle the packet
            _packetHandlers[packetId](packet); 
        }
    });
}   

Update for Memory Tool:

Answer 1

For this one, reported to be with Unity newer versions and stream classes. Here is post that shows the suggestion: https://forum.unity.com/threads/hololens-udp-client-with-datagramsocket-and-memory-leak.583816/

"Stream streamIn = args.GetDataStream().AsStreamForRead();

Massive memory leak in IL2CPP build. Rewriting the message receiving code to not use the Stream class as in the example below eliminates the memory leak, tested in Unity 2018.2.18f1"

Code (CSharp):

using (var reader = args.GetDataReader())
{
            var buf = new byte[reader.UnconsumedBufferLength];
            reader.ReadBytes(buf);
            string message = Encoding.UTF8.GetString(buf);
}