C++ | Group elements of different types without polymorphism in arrays, vectors, sets, etc

Question

I have a class that stores components (a little bit like how an "Entity Component System" in a videogame would) as such:

class Components
{
public:
   
    template <typename T>
    T& AddComponent()
    {
        T* component = new T();
        m_Components.push_back(component);
        return *component;
    }

    //... More Templated code to use the stored components and delete allocated memory

private:
    std::vector<void*> m_Components;
}

I know this works to a point, because I tested it many times.

What I want to know is, if there are any other ways of accomplishing the same, without using polymorphism, that might be safer?

Also, if I want to delete the memory, is it OK to just call delete on a void* that could be pointing to many different types?

Edit: If I have to use polymorphism I will but it would only be an empty class with no data.

The thing is not every instance of my Components class will not always have the same number of components. Imagine I have many different components with different behaviour like 'AudioComponent', 'RenderComponent', etc.

Sometimes an instance of this class will have only one or two components and other times it will have a lot more which is why it makes sense to me to save it in some type of dynamic array.

If there is any other way to accomplish something like this please point me to that direction.