Most efficient way to pass data from C++ to C#

Question

I am looking for the best way to transfer a large amount of data from C++ (struct or a value class?) into a C# class doing as little data copying as possible. In the sample code below, I have a vector of SubClass objects that has the potential to be very large (10+ million). So I want to avoid a data copy if possible.

Should I/can I just allocate the objects in GC first and use them directly in c++ and forget about the native c++ structures? (Performance is my concern with this one.)

Or, is there some trick that I leverage what is allocated in C++ without causing a data copy?

Here is a sample of something along the lines of what I want to use as a transfer between managed and unmanaged code.

#include <string>
#include <vector>
struct SubClass {
    std::string DataItem1;
    // lots more here
    std::string DataItem50;
};

struct Sample {
    int IntValue;
    std::string StringValue;
    std::vector<std::string> SmallList;
    std::vector<SubClass> HugeList;
};

If I can avoid getting into the weeds with pinvoke and COM classes, I would prefer it.

Answer 1

Following the example from Unity (who uses C#), Native plugin example uses a GC handle to transfer data from C# to C++. We can try the opposite to send data to C++ from C#.

Pin down a C# variable to allow faster copying.

using System;
using System.Collections;
using System.Runtime.InteropServices;
// vertices is a Vector3[], where Vector3 is a struct 
// of 3 floats using a sequential layout attribute
void test(){
GCHandle gcVertices = GCHandle.Alloc (vertices, GCHandleType.Pinned); 
}

Transfer the handle to C++ using marshaling. It's unavoidable that you have to copy something. Here copying a pointer should be good enough. More on marshaling according to Microsoft doc.

[DllImport("your dll")]
private static extern void SendHandle(IntPtr vertexHandle, int vertexCount);
SendHandle(gcVertices, vertices.Length);

Inside C++, you'll receive the handle as a pointer type to a C++ type of your choosing. In this case, vertices are a list of structs of 3 floats. The reference code decided to use float *. You just need to do pointer arithmetic properly depending on the pointed type, including the case of void *.

extern "C" __decl(dllexport) void SendHandle(float* vertices, int vertexCount);

Here the example code copies data directly from the pointer, but you can also write to the pointer's location.

for (int i = 0 ; i < vertexCount; i++)
{
   // read from C# heap
   float x = vertices[0];
   float y = vertices[1];
   float z = vertices[2];

   // write to C# heap
   *vertices = sqrt(x);
   *(vertices + 1) = sqrt(y);
   *(vertices + 2) = sqrt(z);

   vertices += 3; // because it is a list of struct of 3 floats
}

Clean up the pinned handle from the C# side to resume the garbage collector.

gcVertices.Free();

As for strings, I believe the interop library has an implementation that handles pointer arithmetic and copying for you. You could probably just use a string type directly inside the exposed export function, as long as you specify how to marshal it with the MarshalAs attribute in C# and a library in C++ if you are not converting to the C type char *.