Unity 3D - Reference DLL in package downloaded via Package Manager

Question

So what I am attempting to do is reference a DLL inside of a custom Package that is downloaded through the Unity Package Manager via GitHub URL.

I can manually add the reference in Visual Studio which is fine until I decide to update the package, then the reference breaks as the package folder name changes. This reference is also only for the path on my machine and wont work with others who would need to re-add the reference.

Currently the package folder structure is Runtime -> Plugins -> library-name.dll and that's it. Essentially I just want a way to distribute at DLL as a package that users can add in Unity / Update to the latest version. The DLL contains classes and functionality that is used both by the Unity game and a webserver (Shared Objects for example).

Is there a way I can reference the DLL in such a way that the reference path is always valid, So even if I update the package the reference doesn't break?

I know unity will automatically search the assets folder for plugins, is there potentially a way to set it to search the package folder as well or have it copy to a folder where unity can pick it up automatically?

Thanks!