Reduce duplication of binary files across C# projects in solution

Question

I am working in a repository with a lot of test projects that include many binary files. A lot of these files are duplicated from project to project, resulting in i) massive duplication in the source code, and ii) massive file-copying and duplication in the build folders. Exemplified it could look like this:

📦Solution
 ┣ 📂Project A
 ┃ ┣ 📂bin
 ┃ ┃ ┗ 📂release
 ┃ ┃ ┃ ┣ 📜BinaryFile1
 ┃ ┃ ┃ ┗ 📜BinaryFile2
 ┃ ┣ 📜BinaryFile1
 ┃ ┗ 📜BinaryFile2
 ┗ 📂Project B
 ┃ ┣ 📂bin
 ┃ ┃ ┗ 📂release
 ┃ ┃ ┃ ┣ 📜BinaryFile1
 ┃ ┃ ┃ ┣ 📜BinaryFile2
 ┃ ┃ ┃ ┗ 📜BinaryFile3
 ┃ ┣ 📜BinaryFile1
 ┃ ┣ 📜BinaryFile2
 ┃ ┗ 📜BinaryFile3

The files in the projects are set up with <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>.

What I am looking for

A way to reduce the amount of duplication in the source code and, if possible, in the build output folders.

What I have tried

I have tried create a separate "data" project containing the binary files and then referencing that project from other projects. That removes the duplication in the source code. However, it results in all the files being copied to the build output for all projects referencing the data-project. This is not desirable.

I have also tried to not reference the data-project from any of the test-projects, and then from the test-projects reference the needed files by adding them with "Add -> Existing item -> Add as link". This also removes the duplication in the source code and in the build output, but it results in the files only being copied to the output of the data-project, and the tests in the test-projects failing because they cannot access the files they need.