OAuth Cognito ID token unauthorized

Question

I have a cognito user pool/app client setup with OAuth flow and standard scopes (email, admin, etc.) and several 3rd party providers (google, azure, etc.). I would like to get the identity data in the request context which mean I would need to use the id token but using the id token gets me an unauthorized response from api gateway while the access token works as expected. I remember reading somewhere that custom scopes doesn't allow for id token auth but I don't have custom scopes setup. Any insight into what's going on here?