Debugging assembly unloadability in .NET 5

Question

I have an assembly which I need to unload dynamically. I have followed the steps in https://docs.microsoft.com/en-us/dotnet/standard/assembly/unloadability for both the development of my loader as well as the debugging using WinDbg. I have a "memory leak" because my unload command is not working. My question is how do I make my unload work?

Initially, I noticed that memory was not going down after using the unload command, so I created a dictionary of weak references which point at the assemblies. I dynamically load and unload the assembly every time I perform an operation, which is creating multiple copies of the assembly in memory. I can use the Dictionary<int, WeakReference> to see if the assemblies have been unloaded or not as they do in the guide above. They aren't getting released, no matter how long I wait, or how many times the garbage gets collected. This says to me that there's some outstanding reference, but for the life of me I can't find it.

I'm not making either of the mistakes they made in the example code- the only reference I hold onto is the weak reference. I even assign the assembly and the assembly loader to null once I'm done with them, though they go out of scope immediately anyway. I don't have any callbacks or, to the best of my knowledge Types across the border because the only communication I do across the boundary is via a file. The assembly is ending appropriately without issues. So I fired up WinDbg and tried to follow their instructions. Everything goes fine until I try to get the GC roots of the AssemblyLoader:

0:000> !gcroot -all 0x00007ff7b943f9c8
Found 0 roots.

And when I step through the threads in my application, I don't see anything that jumps out to me as suspicious, though honestly I don't have enough intuition in this area that that means much. The output is below.

0:021> ~*e !clrstack
OS Thread Id: 0x3510 (0)
        Child SP               IP Call Site
000000FEA5D7E6E0 00007ff8813111c4 [InlinedCallFrame: 000000fea5d7e6e0] Interop+User32.WaitMessage()
000000FEA5D7E6E0 00007ff7b8f38a15 [InlinedCallFrame: 000000fea5d7e6e0] Interop+User32.WaitMessage()
000000FEA5D7E6B0 00007ff7b8f38a15 ILStubClass.IL_STUB_PInvoke()
000000FEA5D7E770 00007ff813db015e System.Windows.Forms.Application+ComponentManager.Interop.Mso.IMsoComponentManager.FPushMessageLoop(UIntPtr, msoloop, Void*) [/_/src/System.Windows.Forms/src/System/Windows/Forms/Application.ComponentManager.cs @ 398]
000000FEA5D7E860 00007ff813db28f5 System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(msoloop, System.Windows.Forms.ApplicationContext) [/_/src/System.Windows.Forms/src/System/Windows/Forms/Application.ThreadContext.cs @ 1111]
000000FEA5D7E8F0 00007ff813db2568 System.Windows.Forms.Application+ThreadContext.RunMessageLoop(msoloop, System.Windows.Forms.ApplicationContext) [/_/src/System.Windows.Forms/src/System/Windows/Forms/Application.ThreadContext.cs @ 975]
000000FEA5D7E950 00007ff813a89c86 System.Windows.Forms.Application.Run(System.Windows.Forms.Form) [/_/src/System.Windows.Forms/src/System/Windows/Forms/Application.cs @ 1198]
000000FEA5D7E990 00007ff7b8bd9f4a ConfigTool.Program.Main() [C:\Users\isherman\Documents\Projects\twmp\FormsConfigTool\Program.cs @ 42]
<snipping out unmanaged threads>
OS Thread Id: 0xa2dc (6)
        Child SP               IP Call Site
000000FEA667FC30 00007ff88338d8c4 [DebuggerU2MCatchHandlerFrame: 000000fea667fc30] 
OS Thread Id: 0xc5fc (7)
        Child SP               IP Call Site
<snip>
OS Thread Id: 0x900c (12)
        Child SP               IP Call Site
<snip>
OS Thread Id: 0x9b88 (16)
        Child SP               IP Call Site
<snip>
OS Thread Id: 0xa358 (18)
        Child SP               IP Call Site
OS Thread Id: 0x9a8c (19)
        Child SP               IP Call Site
<snip>

And here's the code that I use to execute and unload the Assembly Load Context:

        private static Dictionary<int, WeakReference> references;
        private static int refCount =0;
        [MethodImpl(MethodImplOptions.NoInlining)]
        private static int ExecuteAndUnload(string assemblyPath, 
                           XmlDocument doc, out WeakReference alcWeakRef)
        {
            references ??= new Dictionary<int, WeakReference>();
            assemblyPath = Path.GetFullPath(assemblyPath);
            var alc = new SequesteredTagBrowser(assemblyPath);
            Assembly a = alc.LoadFromAssemblyPath(assemblyPath);
            Debug.Assert(a.EntryPoint !=null);
            alcWeakRef = new WeakReference(alc, true);
            references.Add(refCount++, alcWeakRef);
            //I pass some xml in as an argument to the Program.Main function
            var args = new object[] {new[] {doc.OuterXml}};
            a.EntryPoint.Invoke(null, args);
            a = null;

            alc.Unload();
            alc = null;
            GC.Collect();

            return 0;
        }

And here's the code for the SequesteredTagBrowser class:

    public class SequesteredTagBrowser:AssemblyLoadContext
    {
        private AssemblyDependencyResolver _resolver;

        public SequesteredTagBrowser(string mainAssemblyToLoadPath) : base(isCollectible: true)
        {
            
            _resolver = new AssemblyDependencyResolver(mainAssemblyToLoadPath);
        }

        protected override Assembly Load(AssemblyName name)
        {
            string assemblyPath = _resolver.ResolveAssemblyToPath(name);
            if (assemblyPath != null)
            {
                return LoadFromAssemblyPath(assemblyPath);
            }

            return null;
        }

    }