Android adb constantly shows "authorizing" for connected physical device

Question

I am using adb to connect to my physical device. Everything worked for a while, now all of a sudden adb tells me it would be "authorizing" all the time. I have already restarted adb, restarted the physical device, restarted USB debugging on the device but nothing helped.

When I run adb devices it keeps telling me

List of devices attached
42b406de    authorizing

With this, I can not remote debug the device, can not even open the developer menu in the app I want to debug (for opening the developer menu, I use adb shell input keyevent 82 and that returns adb: device still authorizing.

Does anyone have a hint on what to do in such case to get debugging running again?

Thanks and best regards

EDIT: Maybe also important - I am using adb on Mac OS Big Sur