Have to do an additional click on page to interact with it after gesture finish

Question

I am using react-use-gesture in combination with react-spring to implement mobile menu dragging in my website.

The issue is, when I am using drag to close the menu on mobile, it requires an additional click to do something.

Here are the results I got when trying to fix this bug:

1. There are no problems when using PC (though this drag menu is not intended for it)
2. If I wait a few seconds after the drag, you no longer need a click (it is probably because it finishes the slide and removes focus from the element that's being dragged)
3. When you use a close button (whic triggers the same event) there are no problems
4. If your velocity is around zero when you stop dragging there are no problems too

And this is what I've tried:

1. Remove page focus using document.activeElement (that fixed another bug)
2. Try to click on close button explicitly (using .click())

Currently I have this code (it is simplified since I have verified the rest of the website has no impact on the issue):

const [addTagMenuOpen, setAddTagMenuOpen] = useState(false);
/* ... */
const projectTagMenuAStyle = useSpring({
    to: {
        x: addTagMenuOpen ? 1 : 0
    },
    from: {
        x: 0
    }
});
const projectTagMenuAGesture = useDrag(
    ({ last, velocity: [, vy], direction: [, dy], movement: [, my] }) => {
        if (last) {
            (my > window.innerHeight * 0.5 || (vy > 0.5 && dy > 0)) ? TagToggleEvent(null) : projectTagMenuAStyle.x.start(1);
        }
        else projectTagMenuAStyle.x.start(1 - my / window.innerHeight, { immediate: true });
    },
    { from: [projectTagMenuAStyle.x.get(), 0], bounds: { top: 0 }, rubberband: true }
);
/* ... */
const TagToggleEvent = (s: boolean | null) => {
    setAddTagMenuOpen(!addTagMenuOpen);

    if (addTagMenuOpen) {
        (document.activeElement as HTMLElement).blur();
    }
}

And the component part, that is simplified as well:

return /* ... */ <div>
    <div className={styles.addTag} onClick={() => TagToggleEvent(null)}>
        <AddIcon className={styles.addTagIcon} />
    </div>
    <animated.div {...projectTagMenuAGesture()} className={styles.mobileDetailsMenu} style={{ translateY: projectTagMenuAStyle.x.to(x => 100 * (1 - x) + 'vh') }}>
        <CloseIcon onClick={() => TagToggleEvent(null)} className={styles.closeIcon} />
        {/* The div contents (have no impact on the issue) */}
    </animated.div>
</div> /* ... */;