C++ Lua access violation when table size is above treshold

Question

Recently I became Lua expert in my team due to an issue occurring when we want to send a large table to the following function:

  int native_sl_shootlaserpulse(lua_State* L)
  {
    int iRetVal = 0;

    // L1 is class instance
    luaL_checktype(L, 2, LUA_TTABLE);
    double jumpSpeed = luaL_checknumber(L, 3);
    double settleTime = luaL_checknumber(L, 4);
    unsigned int numberOfPulses = luaL_checknumber(L, 5);
    bool simulateLaserPulse = lua_toboolean(L, 6);

    size_t tableSize = lua_objlen(L, 2);

    std::vector<Scriptey::OffsetXY> scanOffsets(tableSize);
    Scriptey::OffsetXY offset;
    bool scanOffsetsValid(true);

    for (size_t i(1); i < tableSize; ++i)
    {
      lua_rawgeti(L, 2, i + 1); // push the value found in the table at L2, index i+1 on the stack
  
      if (lua_istable(L, -1) && lua_objlen(L, -1) == 2)
      {
        lua_pushnil(L);
  
        // Get x
        lua_next(L, -2);
        offset.x = lua_tonumber(L, -1);
        lua_pop(L, 1);
  
        // Get y
        lua_next(L, -2);
        offset.y =lua_tonumber(L, -1);
        lua_pop(L, 1);
  
        scanOffsets[i] = offset;
      }

      else
      {
        printf(
        "Coordinate in table should have exact two values (x and y), actual size: %d\n",
        lua_objlen(L, -1));
        scanOffsetsValid = false;
      }

       lua_pop(L, 1);
    }

    return iRetVal;
  }

This function is meant to convert the Lua table to a C++ vector of a certain type.

In the Lua script the following works fine where X=50:

scanOffsets = {}

for i=1,X do
    scanOffsets[i] = {0.5, 0.6}
end

sl_shootlaserpulse(scanOffsets, 1.0, 1000, 1, 0)

However, when the size of X is increased to 200, the program gives an access violation error. Of course something goes wrong with the memory management in Lua. But I cannot seem to find the actual cause of the crash. The table gets converted correctly. Only when garbage collection is triggered things seem to fall apart.

I tried increasing the stack size in VS linker but that did not work.

Does anyone had similar experiences using Lua with C++?

Answer 1

You have Lua C API stack overflow: each iteration of the loop for (size_t i(1); i < tableSize; ++i) pushes one more value to the API stack.

You can make the loop balanced by inserting lua_pop(L, 1); before scanOffsets[i] = offset; to pop the key pushed by the last call to lua_next(L, -2);.

The better approach is to use lua_rawgeti instead of lua_next to get x and y.

There are two other mistakes in the code:

1. In the loop for (size_t i(1); i < tableSize; ++i) the first element of the array is not being traversed, set initial value of i to 0 to fix it.
2. The Lua number 0 is treated as true by the function lua_toboolean(L, 6);, probably it is not what you want here.

Answer 2

Adding the checkstack call at the top of the loop resolved the read access violations:

for (size_t i(0); i < tableSize; ++i)
{
  luaL_checkstack(L, 3, nullptr); // <-- fix
  lua_rawgeti(L, 2, i + 1); // push the value found in the table at L2, index i+1 on the stack
  
  if (lua_istable(L, -1) && lua_objlen(L, -1) == 2)
  {
    lua_pushnil(L);
  
    // Get x
    lua_rawgeti(L, -2, 2);
    scanOffsets[i].x = lua_tonumber(L, -1);
    lua_pop(L, 1);
  
    // Get y
    lua_rawgeti(L, -2, 1);
    scanOffsets[i].y = lua_tonumber(L, -1);
    lua_pop(L, 1);
  }

  lua_pop(L, 1);
}

Edit: the pushnil call was removed from the loop as rawgeti is being used here. This was the reason a stack overflow occurred. Now the checkstack is also not required and the loop is balanced.