React useEffect adds an event listener to ref on every input change

Question

Take the following component:

function MyComponent() {
    const [ inputValue, setInputValue ] = useState( '' );
    const [ submittedValue, setSubmittedValue ] = useState( '' );
    const inputRef = useRef<HTMLInputElement>( null );

    useEffect( () => {
        const inputElement = inputRef.current;
        const handleEvent = ( event ) => {
            if ( event.key === 'Enter' ) {
                event.preventDefault();
                setSubmittedValue( inputValue );
            }
        };
        console.log( 'adding listener' );
        inputElement?.addEventListener( 'keydown', handleEvent );
        return () => inputElement?.removeEventListener( 'keydown', handleEvent );
    }, [ inputValue ] );

    return (
        <>
            <input value={inputValue} onChange={( e ) => setInputValue( e.target.value )} ref={inputRef}></input>
            <p>{submittedValue}</p>
        </>
    );
}

The component lets you type in a controlled input, and hit the 'Enter' key to "submit" the current input value. The submitted value should display below the input and remain static until "submitting" again.

This works as intended. The problem is that a 'keydown' listener is added on every keystroke.

Expected results: 'adding listener' prints once on mount
Actual results: 'adding listener' prints on every keystroke

I understand why. On every keystroke, onChange runs, which runs setInputValue, which changes useEffect's dependency list, which adds the listener again.

The easiest way to solve this problem is to just add onKeyDown={handleEvent} to the input. But that would be too easy.

I only got to this point because the legacy Input element I'm using does not implement onKeyDown

Is there a way to achieve the expected results without using onKeyDown?