Is it safe to rely on "use_count()" to re-use "shared_ptr" memory in a multithreaded app?

Question

I'm reviewing some old code and I have doubt about the reliability of this one. Unfortunately, the original developer is no longer around so I can't ask him for an explanation.

The following class is in charge of allocating memory on demand for an struct than is used in multiple places and from multiple threads. The getStruct() is called from one place and one thread only, but the returned std::shared_ptr<SomeHeavyStruct> is dispatched to multiple threads. As an optimization, an attempt is made to try to re-use previously created struct instead of throwing away old ones.

#include <vector>
#include <memory>
#include <algorithm>

struct SomeHeavyStruct
{};

class HeavyStructProvider
{
public:
    std::shared_ptr<SomeHeavyStruct> getStruct()
    {
        auto it =
            std::find_if(m_buffer.cbegin(), m_buffer.cend(), [&](const auto& pointer) {
                return pointer.use_count() == 1;
            });

        if (it != m_buffer.cend()) {
            auto pointer = *it;
            return pointer;
        }

        auto newStruct = std::make_shared<SomeHeavyStruct>();
        m_buffer.push_back(newStruct);
        return newStruct;
    }

private:
    std::vector<std::shared_ptr<SomeHeavyStruct>> m_buffer;
};

My concern is that while reading the std::shared_ptr<T>::use_count() documentation, I'm seeing the following:

In multithreaded environment, the value returned by use_count is approximate (typical implementations use a memory_order_relaxed load)

And also:

If use_count returns 1, there are no other owners. [...] In multithreaded environment, this does not imply that the object is safe to modify because accesses to the managed object by former shared owners may not have completed, and because new shared owners may be introduced concurrently, such as by std::weak_ptr::lock.

Knowing that the std::shared_ptr<SomeHeavyStruct> is used from multiple threads, am I right to assume the current implementation is prone to unexpected behavior?

If I had to refactor this code, what would be a suggested approach? Is using a custom Deleter protected by a mutex the way to go (to effectively mark re-usable struct with a custom flag instead of relying on use_count())?