AWS CDK Pipelines using with an existing codepipeline

Question

The documentation of @aws-cdk/pipelines seems to suggest that a CDK pipeline can be added to an existing @aws-cdk/aws-codepipeline/Pipeline, using the codePipeline prop: https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_pipelines.CodePipeline.html

codePipeline? Pipeline An existing Pipeline to be reused and built upon.

However, I am not able to get this to work and am experiencing multiple errors at the cdk synth step, depending on how I try to set it up. As far as I can tell there isn't really any documentation yet to cover this scenario.

Essentially, we are trying to create a pipeline that runs something like:

• clone
• lint / typecheck / unit test
• cdk deploy to test environment
• integration tests
• deploy to preprod
• smoke test
• manual approval
• deploy to prod

I guess it's just not clear the difference between this codebuild pipeline and the cdk pipeline. Also, the naming convention of stages seems a little unclear - referencing this issue: https://github.com/aws/aws-cdk/issues/15945

See: https://github.com/ChrisSargent/cdk-issues/blob/pipelines/lib/cdk-test-stack.ts and below:

import * as cdk from "@aws-cdk/core";
import * as pipelines from "@aws-cdk/pipelines";
import * as codepipeline from "@aws-cdk/aws-codepipeline";
import * as codepipeline_actions from "@aws-cdk/aws-codepipeline-actions";

export class CdkTestStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const cdkInput = pipelines.CodePipelineSource.gitHub(
      "ChrisSargent/cdk-issues",
      "pipelines"
    );

    // Setup the code source action
    const sourceOutput = new codepipeline.Artifact();
    const sourceAction = new codepipeline_actions.GitHubSourceAction({
      owner: "ChrisSargent",
      repo: "cdk-issues",
      branch: "pipelines",
      actionName: "SourceAction",
      output: sourceOutput,
      oauthToken: cdk.SecretValue.secretsManager("git/ChrisSargent"),
    });

    const pipeline = new codepipeline.Pipeline(this, "Pipeline", {
      stages: [
        {
          actions: [sourceAction],
          stageName: "GitSource",
        },
      ],
    });

    const cdkPipeline = new pipelines.CodePipeline(this, "CDKPipeline", {
      codePipeline: pipeline,
      synth: new pipelines.ShellStep("Synth", {
        // Without input, we get: Error: CodeBuild action 'Synth' requires an input (and the pipeline doesn't have a Source to fall back to). Add an input or a pipeline source.
        // With input, we get:Error: Validation failed with the following errors: Source actions may only occur in first stage
        input: cdkInput,
        commands: ["yarn install --frozen-lockfile", "npx cdk synth"],
      }),
    });

    // Produces: Stage 'PreProd' must have at least one action
    // pipeline.addStage(new MyApplication(this, "PreProd"));

    // Produces: The given Stage construct ('CdkTestStack/PreProd') should contain at least one Stack
    cdkPipeline.addStage(new MyApplication(this, "PreProd"));
  }
}

class MyApplication extends cdk.Stage {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StageProps) {
    super(scope, id, props);

    console.log("Nothing to deploy");
  }
}

Any guidance or experience with this would be much appreciated.

Answer 1

First of all, the error Pipeline must have at least two stages is correct. You only got the GitHub checkout/clone command as a single stage. For a second stage, you could use a CodeBuild project to compile/lint/unit test... as you mentioned.

However, what would you like to do with your compiled artifacts then? Build containers to deploy them later? If so, there are better ways with CDK of doing this (DockerImageAsset). This also could save up your preexisting pipeline and you can use the CDK Pipeline directly.

Could you please try to set the property restartExecutionOnUpdate: true, of your regular Pipeline, like in my following snippet?

 const pipeline = new codepipeline.Pipeline(this, "Pipeline", {
      restartExecutionOnUpdate: true,
      stages: [
        {
          actions: [sourceAction],
          stageName: "GitSource",
        },
      ],
    });

This is needed for the self-mutation capability of the CDK pipeline.

Answer 2

I'm able to achieve something similar by adding waves/stages with only pre and post steps into the CDK pipelines, sample code is listed as below, I'm amending on your original code snippet:

import * as cdk from "@aws-cdk/core";
import * as pipelines from "@aws-cdk/pipelines";
import * as codepipeline from "@aws-cdk/aws-codepipeline";
import * as codepipeline_actions from "@aws-cdk/aws-codepipeline-actions";

export class CdkTestStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const cdkInput = pipelines.CodePipelineSource.gitHub(
      "ChrisSargent/cdk-issues",
      "pipelines"
    );

    const cdkPipeline = new pipelines.CodePipeline(this, "CDKPipeline", {
      selfMutation: true,
      crossAccountKeys: true, //can be false if you don't need to deploy to a different account.
      pipelineName,
      synth: new pipelines.ShellStep("Synth", {
        // Without input, we get: Error: CodeBuild action 'Synth' requires an input (and the pipeline doesn't have a Source to fall back to). Add an input or a pipeline source.
        // With input, we get:Error: Validation failed with the following errors: Source actions may only occur in first stage
        input: cdkInput,
        commands: ["yarn install --frozen-lockfile", "npx cdk synth"],
        primaryOutputDirectory: 'cdk.out'
      }),
    });

    // add any additional test step here, they will run parallels in waves
    cdkPipeline.addWave('test', {post: [provideUnitTestStep(this, 'unitTest')]});
    // add a manual approve step if needed.
    cdkPipeline.addWave('promotion', {post: [new ManualApprovalStep('PromoteToUat')]});

    // Produces: Stage 'PreProd' must have at least one action
    // pipeline.addStage(new MyApplication(this, "PreProd"));

    // Produces: The given Stage construct ('CdkTestStack/PreProd') should contain at least one Stack
    cdkPipeline.addStage(new MyApplication(this, "PreProd"));
  }
}

class MyApplication extends cdk.Stage {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StageProps) {
    super(scope, id, props);

    console.log("Nothing to deploy");
  }
}

What's noticing is that you might need to covert the way you write your Codebuild action to the new cdk CodeBuildStep. A sample unit test step could is like below:

const provideUnitTestStep = (
    id: string
): cdkpipeline.CodeBuildStep => {
    const props: CodeBuildStepProps = {
        partialBuildSpec: codebuild.BuildSpec.fromObject({
            version: '0.2',
            env: {
                variables: {
                    DEFINE_VARIBLES: 'someVariables'
                }
            },
            phases: {
                install: {
                    commands: [
                        'install some dependencies',
                    ]
                },
                build: {
                    commands: [
                        'run some test!'
                    ]
                }
            }
        }),
        commands: [],
        buildEnvironment: {
            buildImage: codebuild.LinuxBuildImage.STANDARD_5_0
        }
    };
    return new cdkpipeline.CodeBuildStep(`${id}`, props);
};

It's not so trivial(and straight forward enough) to retrive the underline CodeBuild project Role, you will need to pass in rolePolicyStatements property in the CodeBuildStep props to grant extra permission needed for your test.