How can i obfuscate the call stack in my C program and force the compiler to not push the return address in the stack?

Question

I want to force my generated executable file (for both Linux and Windows), to not push the return address on the stack when i call a function/API.

So what is the easiest way to achieve this? Let's assume i compile the program using GCC or Visual Studio, and i want this to happen for every call instruction. The reason i want to do this is to obfuscate my program.

If doing this is not possible at the compiler level, then how can i achieve the same goal?

For example what about this solution:

I replace every function/API call with a macro that takes the destination function adderss, pushes the arguments, encrypts the return address, pushes the encrypted return address, and does a jmp to the destination function address (using inline assembly ofc) therefore no call instruction is used; then before every return location in my functions, i place a macro that decrypts the pushed return address and jmps to it (again using inline assembly).

So this is the best solution, right?

void foo()
{
    ...
}

int main()
{
    foo(); // obfuscate this in a way that it doesn't use call, or the call somehow doesn't push the return address.
}

Note that i only want to implement this in a way that known debuggers not be able to show the call stack, such as Windbg, xdbg, etc, and that's it. So even a simple xor encryption of return address is suffice for me (And obviously later on if needed, i can make the encryption more complex). But is there any implementation flaw in the proposed solution?

Answer 1

You can't. When you make a call, you need a return address, this can't be in any way modified.

Wanting to use jmp instead is irrelevant. That will work for function A. But for function B, the jmp will be invalid because it will jump back to A again, not to B.

Either you put all your code within a single, ugly, massive function (use #define pseudo-functions to ensure inlining), then you'll have only jmp inside, or you forget to trick the CPU mechanisms.

And even if you write your own compiler, you'll face the exact, same, problem: were should you return after a function is finished, when you have multiple calling points?

---

Obfuscation isn't a good way to protect your code, and it won't stop anybody who knows assembler - because in the end, you must have valid and meaningful ASM instructions to do anything requested. So it can be traced.

You can check this page, or this page, for correct antidebug techniques. But the more tricks like these you put in your code, the more chances you have to be killed instantly by any decent antivirus.

Because don't forget something: in fine, any hacker can replace every annoying part of your code by nop (0x90h in hexa), put the suitable value in the suitable register/memory to bypass everything, or replace any jz/jnz with an inconditional jmp.

If you need something quick and powerful enough to challenge a non-profesional hacker, put a dongle and encrypt your executable through the dongle SDK. Anything else is just a waste of time and, more annoying, a high risk of being detected as malware/virus/trojan.

And doing this cross-platform is even more irrelevant, since most anti-debugging techniques rely on operating system (and sometimes CPU model/revision!!) to work.

And let's admit that you managed to build a perfect antidebug code, with encrypted code and everything possible... What will you do against a Lauterbach probe or other similar hardware debuggers that can't even be detected by your software? It's not everybody's tool, right, but I have several of these beauties at work, for a wide range of CPU... And I'm not the only one.