'Step Function Created using CDK Fails

I have created a state machine using CDK as per the following Python code:

orderbook_upload_state_machine = sfn.StateMachine(
    self, "orderbook_upload_state_machine",
    definition=tasks.LambdaInvoke(
        self, "orderbook_file_splitter_task",
        lambda_function=orderbook_file_splitter
    ).next(
        sfn.Map(self, "orderbook_chunk_processor_map").iterator(
            tasks.LambdaInvoke(
                self, "orderbook_chunk_processor_task",
                lambda_function=orderbook_chunk_processor
            )
        )
    )
)

In the AWS console, I note that the above code has generated the following ASL JSON:

{
  "StartAt": "orderbook_file_splitter_task",
  "States": {
    "orderbook_file_splitter_task": {
      "Next": "orderbook_chunk_processor_map",
      "Retry": [
        {
          "ErrorEquals": [
            "Lambda.ServiceException",
            "Lambda.AWSLambdaException",
            "Lambda.SdkClientException"
          ],
          "IntervalSeconds": 2,
          "MaxAttempts": 6,
          "BackoffRate": 2
        }
      ],
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:eu-west-1:xxxxxxxxxxxx:function:orderbook_file_splitter",
        "Payload.$": "$"
      }
    },
    "orderbook_chunk_processor_map": {
      "Type": "Map",
      "End": true,
      "Iterator": {
        "StartAt": "orderbook_chunk_processor_task",
        "States": {
          "orderbook_chunk_processor_task": {
            "End": true,
            "Retry": [
              {
                "ErrorEquals": [
                  "Lambda.ServiceException",
                  "Lambda.AWSLambdaException",
                  "Lambda.SdkClientException"
                ],
                "IntervalSeconds": 2,
                "MaxAttempts": 6,
                "BackoffRate": 2
              }
            ],
            "Type": "Task",
            "Resource": "arn:aws:states:::lambda:invoke",
            "Parameters": {
              "FunctionName": "arn:aws:lambda:eu-west-1:xxxxxxxxxxxx:function:orderbook_chunk_processor",
              "Payload.$": "$"
            }
          }
        }
      }
    }
  }
}

However, invoking the step function fails with the following error:

{
  "error": "States.Runtime",
  "cause": "Reference path \"$\" must point to array."
}

What gives?

I note that if I tweak the generated ASL in the console as follows, then it's successful. However I wouldn't know how to translate those changes back into the CDK code.

{
  "StartAt": "orderbook_file_splitter_task",
  "States": {
    "orderbook_file_splitter_task": {
      "Next": "orderbook_chunk_processor_map",
      "Retry": [
        {
          "ErrorEquals": [
            "Lambda.ServiceException",
            "Lambda.AWSLambdaException",
            "Lambda.SdkClientException"
          ],
          "IntervalSeconds": 2,
          "MaxAttempts": 6,
          "BackoffRate": 2
        }
      ],
      "Type": "Task",
      "Resource": "arn:aws:lambda:eu-west-1:xxxxxxxxxxxx:function:orderbook_file_splitter"
      // ⭐⭐ I remove the 'Parameters' block and put the function name here 👆
    },
    "orderbook_chunk_processor_map": {
      "Type": "Map",
      "End": true,
      "Iterator": {
        "StartAt": "orderbook_chunk_processor_task",
        "States": {
          "orderbook_chunk_processor_task": {
            "End": true,
            "Retry": [
              {
                "ErrorEquals": [
                  "Lambda.ServiceException",
                  "Lambda.AWSLambdaException",
                  "Lambda.SdkClientException"
                ],
                "IntervalSeconds": 2,
                "MaxAttempts": 6,
                "BackoffRate": 2
              }
            ],
            "Type": "Task",
            "Resource": "arn:aws:states:::lambda:invoke",
            "Parameters": {
              "FunctionName": "arn:aws:lambda:eu-west-1:xxxxxxxxxxxx:function:orderbook_chunk_processor",
              "Payload.$": "$"
            }
          }
        }
      }
    }
  }
}

aws-cdk aws-step-functions

Solution 1:^[1]

I found the solution by following this tutorial which is sort-of related, and I added the payload_response_only attribute to the first lambda task defintion.

orderbook_upload_state_machine = sfn.StateMachine(
    self, "orderbook_upload_state_machine",
    definition=tasks.LambdaInvoke(
        self, "orderbook_file_splitter_task",
        lambda_function=orderbook_file_splitter,
        payload_response_only=True
    ).next(
        sfn.Map(self, "orderbook_chunk_processor_map").iterator(
            tasks.LambdaInvoke(
                self, "orderbook_chunk_processor_task",
                lambda_function=orderbook_chunk_processor
            )
        )
    )
)

The generated ASL now looks like this:

{
  "StartAt": "orderbook_file_splitter_task",
  "States": {
    "orderbook_file_splitter_task": {
      "Next": "orderbook_chunk_processor_map",
      "Retry": [
        {
          "ErrorEquals": [
            "Lambda.ServiceException",
            "Lambda.AWSLambdaException",
            "Lambda.SdkClientException"
          ],
          "IntervalSeconds": 2,
          "MaxAttempts": 6,
          "BackoffRate": 2
        }
      ],
      "Type": "Task",
      "Resource": "arn:aws:lambda:eu-west-1:xxxxxxxxxxxx:function:adam-hyperchain-orderbook_file_splitter"
    },
    "orderbook_chunk_processor_map": {
      "Type": "Map",
      "End": true,
      "Iterator": {
        "StartAt": "orderbook_chunk_processor_task",
        "States": {
          "orderbook_chunk_processor_task": {
            "End": true,
            "Retry": [
              {
                "ErrorEquals": [
                  "Lambda.ServiceException",
                  "Lambda.AWSLambdaException",
                  "Lambda.SdkClientException"
                ],
                "IntervalSeconds": 2,
                "MaxAttempts": 6,
                "BackoffRate": 2
              }
            ],
            "Type": "Task",
            "Resource": "arn:aws:states:::lambda:invoke",
            "Parameters": {
              "FunctionName": "arn:aws:lambda:eu-west-1:xxxxxxxxxxxx:function:adam-hyperchain-orderbook_chunk_processor",
              "Payload.$": "$"
            }
          }
        }
      }
    }
  }
}

EDIT: This change (output_path) also solves the problem:

orderbook_upload_state_machine = sfn.StateMachine(
    self, "orderbook_upload_state_machine",
    definition=tasks.LambdaInvoke(
        self, "orderbook_file_splitter_task",
        lambda_function=orderbook_file_splitter,
        output_path="$.Payload"
    ).next(
        sfn.Map(self, "orderbook_chunk_processor_map").iterator(
            tasks.LambdaInvoke(
                self, "orderbook_chunk_processor_task",
                lambda_function=orderbook_chunk_processor
            )
        )
    )
)

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1

'Step Function Created using CDK Fails

Solution 1:[1]

Sources

Related Questions

Solution 1:^[1]