Framer Motion - How to delay just the first animation

Question

Is there anyway to delay an animation only for the first animation?

For fun I'm trying to animate a button whose background moves only after 1 second of mousing over the button to look like an eye following the cursor.

I'd like for it to wait 1 second, then animate like normal.

Right now if I set a delay, the button waits a second at a time before following the cursor which isn't what I am going for.