Have you ever tried calling a C# method returning a structure from C++? This is not a common use of interoperability so you probably haven’t, but let me tell you a story about shooting yourself in the foot.

Imagine the following scenario: you have a C++ library that exports an interface of callback methods that will be called by the library during the lifetime of the application. One of the interface’s methods returns a struct (let’s call them GetStruct() and SimpleStruct, respectively). So far so good. Now you want to port your library for .NET by making a wrapper of the C++ library. You make a managed counterpart of the C++ interface and SimpleStruct (with all the marshaling needed, if any) and you’re done. Except that it doesn’t(always) work.

Here’s an example of a case when it doesn’t. Let this be our C++ library (the interface part I was talking about is omitted for brevity)

Nothing special about this code, except some unnecessary typedefs and weird processing of the returned value in FireCallback but I’ll get to that in a second. This will be the C# part of the program:

Again,some standard interop use. If you compile the library and executable you’d expect a 3 written in the console, but instead you get this:

Wait, what, stack corruption? Everything’s fine before the call of g_MyCallback(x, y) but it somehow corrupts the stack. If you add a breakpoint in MakeResult in the C# code you’ll notice something interesting.

The value of x is something funny and y is 1 instead of 2. It seems like the parameters are offset by one. And yet all the other methods of our imaginary interface returning primitive types work? This calls for some disassembly. Let’s see what happens in g_MyCallback(x, y).

Ok, we push the x and y parameters and then push something else. The theory for offsetting the parameters by one seems correct. But why is the compiler doing this? Well, our method returns a struct by value and copying it isn’t very effective, so the (Named) Return Value Optimization kicks in (it’s applied even when compiling with /Od). In short, the last pushed parameter is the address where the returned value will be stored and no copying will occur. The C++ compiler is aware of this fact and works its magic. When we cross the language boundary to C#, however, the stack is broken. You’d expect that the CLR would know these things, and it does, but we hit a corner case. The rules for function return values can be found here. More specifically:

• POD return values 32 bits or smaller will be returned in the EAX register.
• POD return values 33-64 bits in size will be returned via the EAX:EDX registers.
• Non-POD return values or values larger than 64-bits, the calling code will allocate space and passes a pointer to this space via a hidden parameter on the stack. The called function writes the return value to this address.

(The bullets are points 12,13 and 14)

The C# compiler simply couldn’t know if the C++ structure is a POD or not so it applies the rules for non-PODs and it doesn’t expect the hidden parameter. With the mystery unveiled, we have the following options for making our scenario work:

• Make the structure a POD. In the example we can do this by removing the constructor. It’s the only thing breaking the POD-ness.
• Change the signature of the callback in the C++ code so it returns an integral type of the same size. In other words, change “typedef SimpleStruct ReturnType” to “typedef int ReturnType”. This way the compiler won’t emit code for RVO. If you have a 64-bit structure, you can use long long.
• Instead of return value, make the structure an output parameter.
• Add bogus fields in the structure to make it larger than 64-bits.

The last option is the least desirable one and I added it for completeness. Since having a constructor is useful in some cases, I opted for the signature change in our project which lets us keep the non-POD parts and is hidden from the user. It’s not the prettiest solution and you have to keep it in mind if you ever change the size of the structure but it works :).

Note that for x64 builds the stack won’t be corrupted (when compiling with Visual Studio) because the first four integral or pointer parameters are saved in the RCX, RDX, R8 and R9 registers. In the example function we only have 2 parameters, so the hidden RVO parameter will go to a register (if the function had 4 or more arguments, then maybe we will corrupt the stack, if the compiler decides to push the additional arguments and not preallocate memory by modifying the stack pointer at the beginning of the function). The return value will be wrong though, because the compiler will generate code that interprets the returned value in RAX as an address, and not a value, so it will read the memory at that address. This can be fixed using the same solutions as the ones for a 32-bit build.

Follow Nick on Twitter: @Nikxio