AWS sam deploy hangs on pretraffic hook

Question

I'm learning AWS SAM (serverless) and trying to configure a pretraffic hook function that would do things like database migration. I'm following this example in JavaScript:

docs: https://github.com/aws/serverless-application-model/blob/master/docs/safe_lambda_deployments.rst

JavaScript handler: https://github.com/aws/serverless-application-model/blob/d168f371f494196a57032313075db9faae5587e4/examples/2016-10-31/lambda_safe_deployments/src/preTrafficHook.js

It's hanging when it gets to the part where it would call the pretraffic hook, but never invokes it. The hook function name starts with CodeDeployHook_ so the role has permissions to invoke it. (I know because I previously missed this and got a permissions error.)

Is there any way to see where and why it's hanging? I don't see this in the CodeDeploy console, it just gets stuck at "Pre-deployment validation." (It says 50% complete during deploy, but when I stop and rollback it shows only 1% complete).

I can manually invoke the function via the Test tab of the console.

There could be an issue with my function code, as there is no handler interface to implement for CodeDeploy hooks that would ensure the correct signature. But CodeDeploy SDK does have objects for the request/response that match what is in the JS file. Here is my code:

class PreTrafficFunction implements RequestHandler {

    private static final Logger logger = LoggerFactory.getLogger(PreTrafficFunction.class);

    /**
     * Performs logic required before traffic is routed to a Lambda
     *
     * @param input   a PutLifecycleEventHookExecutionStatusRequest
     * @param context The Lambda execution environment context object.
     * @return The Lambda Function output
     */
    public Object handleRequest(Object input, Context context) {
        PutLifecycleEventHookExecutionStatusRequest event = (PutLifecycleEventHookExecutionStatusRequest) input;
        logger.info(event.toString());
        AmazonCodeDeploy cd = AmazonCodeDeployClientBuilder.standard().build();
        return cd.putLifecycleEventHookExecutionStatus(event.withStatus("Succeeded"));
    }
}

template.yaml:

Parameters:
  # Only reasonable way to have multiple environments is entirely separate CFN stacks for each
  Env:
    Type: String
    Default: Dev
    AllowedValues:
      - Dev
      - Prod
      - Local
    Description: Enter Local, Dev, or Prod - Dev is the default.

Globals:
  Api:
    # This is only necessary to prevent SAM from creating an unnecessary stage named "Stage" that you have to delete later.
    OpenApiVersion: 3.0.1
  Function:
    Timeout: 300
    Runtime: java11
    Architectures:
      - x86_64
    Environment: # More info about Env Vars: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#environment-object
      Variables:
        APP_ENV: !Ref Env

Resources:
  # Create one API Gateway that the rest use
  RestApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: !Ref Env

  DBTesterFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: LambdaZipFunction
      Handler: lambdazip.DBTester::handleRequest
      AutoPublishAlias: !Ref Env
      MemorySize: 512
      Events:
        DBTester:
          # API Gateway: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-api.html
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /db
            Method: get
            RestApiId:
              Ref: RestApi
      DeploymentPreference:
        # Point here is to run pre-traffic function to create/migrate DB tables before traffic hits
        Type: AllAtOnce
        Hooks:
          # Validation Lambda functions that are run before & after traffic shifting
          PreTraffic: !Ref PreTrafficFunction

  PreTrafficFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: LambdaZipFunction
      Handler: lambdazip.PreTrafficFunction::handleRequest
      MemorySize: 512
      FunctionName: 'CodeDeployHook_preTrafficHook'
      DeploymentPreference:
        Enabled: False
        Role: ""
      Policies:
        - Version: "2012-10-17"
          Statement:
            - Effect: "Allow"
              Action:
                - "codedeploy:PutLifecycleEventHookExecutionStatus"
              Resource:
                !Sub 'arn:${AWS::Partition}:codedeploy:${AWS::Region}:${AWS::AccountId}:deploymentgroup:${ServerlessDeploymentApplication}/*'
        - Version: "2012-10-17"
          Statement:
            - Effect: "Allow"
              Action:
                - "lambda:InvokeFunction"
              Resource: !GetAtt DBTesterFunction.Arn
      Environment:
        Variables:
          CurrentVersion: !Ref DBTesterFunction.Version

Answer 1

Few things got in the way of this.

First, as mentioned I initially omitted the function name, which is required by the IAM role. For that, I saw an error in the CodeDeploy console for the deploy.

I added this and the permissions error went away, but it still did not invoke the function (or didn't appear to). This is where it hung as described above.

I had to comment out the hook section and redeploy to allow that part of the change set to finish and change the name of the function. Adding the hook section back, I could now see CloudWatch logs for that function. It appears it tries a few times and stops, or perhaps it is using the standard decaying retries.

I added log messages to show the class and properties of the input object. Turns out, it's not being marshalled to a PutLifecycleEventHookExecutionStatusRequest (which would make the most sense) or even a normal object with fields, but rather a LinkedHashMap. Knowing that, I could process the input to create the status request and submit it to CodeDeploy. Here is my working class in Groovy - trivial to convert to Java:

@CompileStatic
class PreTrafficFunction implements RequestHandler {

    private static final Logger logger = LoggerFactory.getLogger(PreTrafficFunction.class)

    /**
     * Performs logic required before traffic is routed to a Lambda.
     *
     * @param input a Map with the fields of a PutLifecycleEventHookExecutionStatusRequest
     * @param context The Lambda execution environment context object.
     * @return PutLifecycleEventHookExecutionStatusResult
     */
    Object handleRequest(Object input, Context context) {
        logger.info("Input properties:\n${input.properties}\nInput:\n${input?.toString()}")
        Map<String, String> event = input as Map<String, String>
        if (!input) {
            throw new IllegalArgumentException("Input is null or empty")
        }
        PutLifecycleEventHookExecutionStatusRequest statusRequest = new PutLifecycleEventHookExecutionStatusRequest()
                .withDeploymentId(event.DeploymentId)
                .withLifecycleEventHookExecutionId(event.LifecycleEventHookExecutionId)
                .withStatus('Succeeded')
        AmazonCodeDeploy cd = AmazonCodeDeployClientBuilder.standard().build()
        cd.putLifecycleEventHookExecutionStatus(statusRequest)
    }
}