Prevent javascript throttle function repeating if user pauses on trigger

Question

I want to resize a header when the user scrolls, by adding/subtracting a class. But, I'm missing something. The script below fires repeatedly, re-adding the class, If a user slowly scrolls down the page to the offset point that triggers the function. It doesn't matter what time I set, the user has still scrolled to that location and stopped - triggering the script repeatedly. I tried with a basic debouncer, and that didn't work - I got similar issues.

Here's my script:

let throttlePause;
 
const throttle = (callback, time) => {
  if (throttlePause) return;
  throttlePause = true;
  setTimeout(() => {
    callback();
    throttlePause = false;
  }, time);
};

 
function scrollShrink() {
  document.getElementById('wrapper').classList.toggle('page-scrolled', window.pageYOffset >= 20);
}

// run through throttler
window.addEventListener("scroll", () => { 
  throttle(scrollShrink, 200);
});