Pointer events firing at the wrong time

Question

I am running into a problem where mousing over one object in my scene causes all of the onPointerOver listeners in the scene to fire at once, sometimes randomly. I think this is because of the way that I am adding components to the scene, but I'm not sure.

What could be causing this behavior? Here is an example of my code for reference.

I have a SampleObject constructor like this, which takes an image path and a mouseOver function as props:

const SampleObject = ({ image, mouseOver }) => {
    const ref = useRef();
    const texture = useLoader(TextureLoader, image);
    return (<mesh ref={ref} onMouseOver={mouseOver}>
       <planeBufferGeometry attach="geometry" />
       <meshPhongMaterial attach="material" map={texture}/>
    </mesh>)
}
export default React.memo(SampleObject);

And in the render, I am loading it using Suspense because it involves loading an image. The onPointerOver listener updates a state:

<Suspense fallback={null}>
    <SampleObject image={ images(`./image.png`).default } onPointerOver={() => {  setMiscState(true)  }} />
    <SampleObject image={ images(`./image.png`).default } onPointerOver={() => {  setMiscState(false)  }} />
</Suspense>

Therefore, when the state changes, all of these SampleObject components re-render. Is that what's causing the onPointerOver events to fire at the wrong times?