C++ Efficiently Moving Object Created Via Placement New In Raw Memory Buffer

Question

After a fair amount of googling I haven't found a clear answer to this question. I am writing a custom memory pool for a project I am working. You can think of this as a sparse set mapping to a tightly packed data array. The code works as follows:

There is a vector of raw memory 'chunks' each with a size that is a multiple of the stored object's size.

sparseIndices maps a large set of potential indices to a small set in packedIndices where each stores an index pointing to the other.

index            0   1   2
memory         | a | b | c |
                 |   |   |
                 |   |   |                       
packedIndices  | 3 | 4 | 1 |             // indices in the sparse set

sparseIndices  | - | 2 | - | 0 | 1 |     // indices in the packed set
index            0   1   2   3   4

Objects are created using code like the code listed below. I'ved ommited the parts where the index vectors are updated.

// chunkSize usually much larger than 1 (i.e. 8192)
uint8_t* chunk = new uint8_t[chunkSize * sizeof(T)];
memory.push_back(chunk);

// new objects are always placed at the end of 'memory' so every valid object lies in contiguous memory
// there is always enough space for another object since I allocate memory as required.
// the index arrays are modified near this code to have the correct mappings.
T* myT = new (memory[chunkIndex] + offset)T();

vector<uint8_t*> memory;
vector<size_t> packedIndices
vector<size_t> sparseIndices;

When an object needs to be removed I would like to "swap" the object with the last object, delete the last object (now the one to be deleted) and then resize the memory 'pool' - 1.

My question is, what is the proper way to "swap" the two objects? So, far I have been using memcpy & memset as illustrated below. This works, but, I have a feeling this is A.) not good practice B.) flat out wrong for complex types. I considered template polymorphism being an issue but, from my limited understanding, this doesn't seem possible since C++ doesn't seem to support covariant templates (C++ covariant templates).

  // the goal here is to keep the objects tightly "packed" in contiguous memory
  T* element = static_cast<T*>(this->GetRawPointer(memoryIndexToRemove));
  element->~T();
  void* lastElement = this->GetRawPointer(lastValidMemoryIndex);
  memcpy(static_cast<void*>(element), lastElement, sizeof(T));
  memset(lastElement, 0, sizeof(T));
  // the size will be modified here so the previous last object is now free memory