How to stop script re-firing add/subtraction of class, if user pauses scrolling at pageYOffset

Question

I have a javascript to collapse a header when the user scrolls down the page - and to re-expand it if they scroll back to the top. It works fine for strong, decisive scrolling. But if a user pauses at or close to the pageYOffset trigger, the header 'flickers' - because the script removes and adds the 'page-scrolled' class over and over.

Maybe I need a class listener to temporarily turn off the script? But, in an earlier version of the script where I added a listener, the script only triggered once. If the user scrolled back up the page, the 'page-scrolled' class wasn't removed if the user scrolled back up - because the script was no longer listening. As I'm not well-versed on javascript, I'm not sure how to solve this, or if my syntax and location are correct. What's a best solution?

Here's what I have:

function scrollShrink() {

  var elementTarget = document.querySelector('.header');
  
    if(window.pageYOffset >= 20) {
        document.querySelector('.header').classList.add('page-scrolled');
        // Below 'remove' line didn't work - stops function entirely, which isn't wanted.
        // document.removeEventListener('scroll', scrollShrink);
        
    } else {
        document.querySelector('.header').classList.remove('page-scrolled');
    }
  };

document.addEventListener('scroll', scrollShrink);