Why is saving a ID3D11Texture2D to file so painfully slow and what to do about it

Question

I'm having some code (compiled into a standalone executable) to take a screenshot. It runs pretty fast (around 100ms), however, as soon as I run something graphic intensive (like a game), it takes up to 2 minutes, even though CPU and GPU are not fully exhausted. Specifically, I use IDXGIOutputDuplication to grab the screen, and then some code I have "borrowed" from here https://github.com/microsoft/DirectXTex/blob/main/ScreenGrab/ScreenGrab11.cpp to save the screen as a picture to file. Specifically the "encoding" part, i.e., use of the encoder seems to be problem. Any ideas about (1) where this massive performance impact is coming from and (2) how to improve it?

Full code here: https://github.com/plengauer/DXGIOutputDuplication