Why does Cognito IDP initiate_auth not authorise API Gateway in this setup?

Question

I have a cloud formation stack containing a Cognito User Pool and its client, an API Gateway and an Authorizer.

If I log in using the hosted form I get JWT tokens - and the access_token successfully authorises the API.

However if I log using idp.initiate_auth(), using the same credentials, I also get a set of JWT tokens - but the access_token won't allow me to access the API.

I can see that the hosted form does some opaque magic inside a function called getAdvancedSecurityData() and writes a form variable called cognitoAsfData... but beyond that I cannot make sense of it.

Here is - I hope the relevant sections of CFN config:

AWSTemplateFormatVersion: "2010-09-09"
Description: ""
Resources:
  LADApiGatewayRestApi:
    Type: "AWS::ApiGateway::RestApi"
    Properties:
      Name: "DemoApi"
      Description: "This is the demo API."
      ApiKeySourceType: "HEADER"
      EndpointConfiguration:
        Types:
        - "REGIONAL"

  LADApiGatewayStage:
    Type: "AWS::ApiGateway::Stage"
    Properties:
      StageName: "test"
      DeploymentId: !Ref LADApiGatewayDeployment
      RestApiId: !Ref LADApiGatewayRestApi
      CacheClusterEnabled: false
      TracingEnabled: false

  LADApiGatewayDeployment:
    Type: "AWS::ApiGateway::Deployment"
    Properties:
      RestApiId: !Ref LADApiGatewayRestApi
    DependsOn:
      - LADApiGatewayResource
      - LADApiGatewayMethod
      - LADApiGatewayRestApi

  LADApiGatewayResource:
    Type: "AWS::ApiGateway::Resource"
    Properties:
      RestApiId: !Ref LADApiGatewayRestApi
      PathPart: "transactions"
      ParentId: !GetAtt LADApiGatewayRestApi.RootResourceId

  LADApiGatewayMethod:
    Type: "AWS::ApiGateway::Method"
    Properties:
      RestApiId: !Ref LADApiGatewayRestApi
      ResourceId: !Ref LADApiGatewayResource
      HttpMethod: "GET"
      AuthorizationType: "COGNITO_USER_POOLS"
      AuthorizerId: !Ref LADApiGatewayAuthorizer
      ApiKeyRequired: false
      MethodResponses:
      -
        ResponseModels:
          "application/json": "Empty"
        StatusCode: "200"
      Integration:
        CacheNamespace: !Ref LADApiGatewayResource
        ContentHandling: "CONVERT_TO_TEXT"
        IntegrationHttpMethod: "POST"
        IntegrationResponses:
        -
          ResponseTemplates: {}
          StatusCode: "200"
        PassthroughBehavior: "WHEN_NO_MATCH"
        TimeoutInMillis: 29000
        Type: "AWS"
        Uri: !Sub "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${LADLambdaFunction}/invocations"
      AuthorizationScopes:
      - "email"

  LADApiGatewayAuthorizer:
    Type: "AWS::ApiGateway::Authorizer"
    Properties:
      RestApiId: !Ref LADApiGatewayRestApi
      Name: "LADDemoApiAuthorizer"
      Type: "COGNITO_USER_POOLS"
      ProviderARNs:
      - !GetAtt LADCognitoUserPool.Arn
      AuthType: "cognito_user_pools"
      IdentitySource: "method.request.header.Authorization"
  
  LADCognitoUserPool:
    Type: "AWS::Cognito::UserPool"
    Properties:
      UserPoolName: "DemoApp"
      Policies:
        PasswordPolicy:
          MinimumLength: 8
          RequireUppercase: true
          RequireLowercase: true
          RequireNumbers: true
          RequireSymbols: true
          TemporaryPasswordValidityDays: 7
      LambdaConfig: {}
      Schema:
      -
        Name: "sub"
        (...)
      AutoVerifiedAttributes:
      - "email"
      MfaConfiguration: "OFF"
      EmailConfiguration:
        EmailSendingAccount: "COGNITO_DEFAULT"
      AdminCreateUserConfig:
        AllowAdminCreateUserOnly: false
      UserPoolTags: {}
      AccountRecoverySetting:
        RecoveryMechanisms:
        -
          Priority: 1
          Name: "verified_email"
        -
          Priority: 2
          Name: "verified_phone_number"
      UsernameConfiguration:
        CaseSensitive: false
      VerificationMessageTemplate:
        DefaultEmailOption: "CONFIRM_WITH_CODE"

  LADCognitoUserPoolClient:
    Type: "AWS::Cognito::UserPoolClient"
    Properties:
      UserPoolId: !Ref LADCognitoUserPool
      ClientName: "DemoAppClient"
      RefreshTokenValidity: 30
      ReadAttributes:
      - "address" (...)
      ExplicitAuthFlows:
      - "ALLOW_CUSTOM_AUTH"
      - "ALLOW_REFRESH_TOKEN_AUTH"
      - "ALLOW_USER_SRP_AUTH"
      - "ALLOW_USER_PASSWORD_AUTH"
      GenerateSecret: false
      PreventUserExistenceErrors: "ENABLED"
      SupportedIdentityProviders:
      - "COGNITO"
      CallbackURLs:
      - "https://example.com/callback"
      LogoutURLs:
      - "https://example.com/signout"
      AllowedOAuthFlows:
      - "code"
      - "implicit"
      AllowedOAuthScopes:
      - "aws.cognito.signin.user.admin"
      - "email"
      - "openid"
      - "phone"
      - "profile"
      AllowedOAuthFlowsUserPoolClient: true
      IdTokenValidity: (...)

  LADCognitoUserPoolDomain:
    Type: "AWS::Cognito::UserPoolDomain"
    Properties:
      Domain: !Sub "ladauthdemo${LADApiGatewayStage}"
      UserPoolId: !Ref LADCognitoUserPool

In the case of access_token from the hosted html I get tokens which decode like this:

{
  "kid": "jFXB1AW4cAjbF1Ti+Tru/8ToxbYCAB1IYdCwEGfM7Sk=",
  "alg": "RS256"
}
{
  "sub": "eabb63bf-7bf2-464f-9d94-808239738981",
  "iss": "https://cognito-idp.eu-west-2.amazonaws.com/eu-west-2_EyTaD2AsF",
  "version": 2,
  "client_id": "40vstm6vrhas7kokoasnrhf1g1",
  "event_id": "2cbcc84d-db84-4ab5-91db-4088cd18829a",
  "token_use": "access",
  "scope": "aws.cognito.signin.user.admin phone openid profile email",
  "auth_time": 1652623673,
  "exp": 1652627273,
  "iat": 1652623673,
  "jti": "84921f7a-bd8e-4fea-a89d-984b1645b6e2",
  "username": "[email protected]"
}

In the case of access_token from the api only the header part is base64 decodable. The header reads the same as the header above.

Here's an example of a whole encoded token from the api:

eyJraWQiOiJqRlhCMUFXNGNBamJGMVRpK1RydVwvOFRveGJZQ0FCMUlZZEN3RUdmTTdTaz0iLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiI1ZTJjYTRlOC0wYzNlLTQwZGYtYWJkMS0wOGFmOGI3MmM0YTIiLCJpc3MiOiJodHRwczpcL1wvY29nbml0by1pZHAuZXUtd2VzdC0yLmFtYXpvbmF3cy5jb21cL2V1LXdlc3QtMl9FeVRhRDJBc0YiLCJjbGllbnRfaWQiOiI0MHZzdG02dnJoYXM3a29rb2FzbnJoZjFnMSIsIm9yaWdpbl9qdGkiOiI4NDAyMjc4ZC0xMzA4LTRmZTgtODUyNC1kNjcxNjliNTg0ZjciLCJldmVudF9pZCI6IjdjMDA2YTQ1LWZiOTUtNDU1Mi05MWNhLTk2YTUwZjQ5MjRmOSIsInRva2VuX3VzZSI6ImFjY2VzcyIsInNjb3BlIjoiYXdzLmNvZ25pdG8uc2lnbmluLnVzZXIuYWRtaW4iLCJhdXRoX3RpbWUiOjE2NTI2MzgwMDgsImV4cCI6MTY1MjY0MTYwOCwiaWF0IjoxNjUyNjM4MDA4LCJqdGkiOiIwNjNlMzRhOC0wYzQ5LTRlNzItYTBjYi0wYjRhZTcxMTdhYmIiLCJ1c2VybmFtZSI6InRlc3R1c2VyLTM2MDkzZGZiLTViNWMtNGFkNi1hMzk4LWIwODQ2ODdlODUzN0B0dW5hc29uaXEuaW8ifQ.VQTz1cFOKUfn-QbOWfl4lk7gzbmptKS-tiCWlBAFbagMt4NvhAZVaLg_Sh7N8BNKw1apkjgQVeC0Coq4otQcDp04z4dt4kyvjFGUO7hdHbgDpp6vNL5pk3yy48a8Zw011gz9fZaAE9CcebBjLFWsWNix8JrN86CwtRiXcoBq6aRKeJez_HVPdLoT8cebESnmI5KbR7DeLcv_J7S-t5DJ9V__X9ksF8yybKlgiytBH7F9RvmCyMH3-8vuxLxOd8ZJ7MOx8NgXKjTw5ETkAWWWGSwp8FBw8Npkf2KmHiRMPaYjg-_sAJPirsc4tyBG8fDhwNCjnEFrj6VfzBjHxgadSA

Answer 1

Your API token is missing the email scope, which is required to access the APIG resource. You need to request the scopes which you want the token to be issued for when you make the call to the API, or you can manually add them with a Cognito pre-token-generation lambda.

Answer 2

There is no way that I've been able to discover, of getting back scopes other than 'aws.cognito.signin.user.admin' using the published APIs. The only way to get those scopes is to leverage the opaque magic in the hosted HTML from Cognito.

However you can add 'aws.cognito.signin.user.admin' to the AuthorizationScopes on your AWS::ApiGateway::Method and it works.

I think this a bit of an abuse of OAUTH process so don't use it to handle access to the nuclear codes.