Hook library function using custom method using dlsym

Question

I am recently started looking in hooking into library from C++ code.
There is a slight confusion with the symbol table creation.
Below is my code (Picked from some online resource, I compiled C code with C++)

hook_main.cpp

#include <stdio.h>
#include <malloc.h>
#include <stdlib.h>

int main()
{
  int *p;
  p = (int *) malloc(10);
  free(p);
  return 0;
}

hook_lib.cpp

#include <stdio.h>
#include <stdint.h>
#include <dlfcn.h>

void *malloc(size_t _size)
{
  static void* (*my_malloc)(size_t) = NULL;
  printf("Custom malloc called\n");
  if(!my_malloc)
    my_malloc = dlsym(RTLD_NEXT,"malloc");
  void *p = my_malloc(_size);
  return p;
}

I am compiling both the files using c++, however it doesn't give the desired output. While debugging, I added

#include <iostream>

in hook_lib.cpp and suddenly my symbol table got changed (library started showing the definition of malloc)

Can somebody please put some light on this behavior. Is this something to do with name mangling ?

Answer 1

Your hook_lib.cpp doesn't compile. It could be something like this:

#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>

void *malloc(size_t _size)
{
  static void *(*real_malloc)(size_t) = NULL;
  printf("Custom malloc called\n");
  if(!real_malloc)
    *(void **)real_malloc = dlsym(RTLD_NEXT,"malloc");
  void *p = real_malloc(_size);
  return p;
}

Answer 2

It is happening because of name mangling by C++.

The function name in object file get modified to _Z6malloc which is the mangled C++ name. Now, when I included iostream, maybe it included the chain of headers which provided the extern declaration of malloc.

Essentially, we get the same expected behavior if we just declare

extern "C"
{
  void *malloc(size_t);
}

in hook_lib.cpp

If we inspect the object file after this, the function name stays as malloc and dlsym is able to locate our function.