AWS Code Build - cached DOWNLOAD_SOURCE taking too long

Question

I am trying to improve the build speed of one of my projects on CodeBuild. The project uses a Github source provider and source caching of type local is enabled.

The first time I ran the build it took 103 secs. I ran it again immediately after the first one finished expecting it to run in a few seconds due to source caching, but it took 60 secs.

What I am missing here? Is the cache not working? If it is working, why does it take that long on the second run?

Thanks

Project Details:

{
    "projectsNotFound": [],
    "projects": [
        {
            "environment": {
                "computeType": "BUILD_GENERAL1_LARGE",
                "imagePullCredentialsType": "SERVICE_ROLE",
                "privilegedMode": true,
                "image": "111669150171.dkr.ecr.us-east-1.amazonaws.com/***********/ep-build-env:latest",
                "environmentVariables": [
                    {
                        "type": "PLAINTEXT",
                        "name": "NEXUS_URI",
                        "value": "http://***************"
                    },
                    {
                        "type": "PLAINTEXT",
                        "name": "REGISTRY",
                        "value": "111669150171.dkr.ecr.us-east-1.amazonaws.com/*********"
                    }
                ],
                "type": "LINUX_CONTAINER"
            },
            "timeoutInMinutes": 60,
            "name": "StorefrontApi",
            "serviceRole": "arn:aws:iam::111669150171:role/CodeBuild-ECRReadOnly",
            "tags": [],
            "artifacts": {
                "type": "NO_ARTIFACTS"
            },
            "lastModified": 1571227097.581,
            "cache": {
                "type": "LOCAL",
                "modes": [
                    "LOCAL_DOCKER_LAYER_CACHE",
                    "LOCAL_SOURCE_CACHE",
                    "LOCAL_CUSTOM_CACHE"
                ]
            },
            "vpcConfig": {
                "subnets": [
                    "subnet-fd7f958b"
                ],
                "vpcId": "vpc-71e3f414",
                "securityGroupIds": [
                    "sg-19b65e6c",
                    "sg-9e28e9f9"
                ]
            },
            "created": 1571082681.262,
            "sourceVersion": "refs/heads/ep-mysql",
            "source": {
                "buildspec": "version: 0.2\n\nphases:\n  build:\n    commands:\n      - env\n      - cd extensions\n      - mvn --settings $CODEBUILD_SRC_DIR_DEVOPS_WINE/pipelines/storefront/build-war/settings.xml --projects storefront/ext-storefront-webapp -am -DskipAllTests clean install\n\nartifacts:\n  secondary-artifacts:\n    storefront-war:\n      base-directory: $CODEBUILD_SRC_DIR/extensions/storefront/ext-storefront-webapp/target\n      files:\n        - \"*.war\"\n\ncache:\n  paths:\n    - '/root/.m2/**/*'\n    - '/root/.npm/**/*'",
                "insecureSsl": false,
                "gitSubmodulesConfig": {
                    "fetchSubmodules": false
                },
                "location": "https://github.com/*****************.git",
                "gitCloneDepth": 1,
                "type": "GITHUB",
                "reportBuildStatus": false
            },
            "badge": {
                "badgeEnabled": false
            },
            "queuedTimeoutInMinutes": 480,
            "secondaryArtifacts": [],
            "logsConfig": {
                "s3Logs": {
                    "status": "DISABLED",
                    "encryptionDisabled": false
                },
                "cloudWatchLogs": {
                    "status": "ENABLED"
                }
            },
            "secondarySources": [
                {
                    "insecureSsl": false,
                    "gitSubmodulesConfig": {
                        "fetchSubmodules": false
                    },
                    "location": "https://github.com/*****************.git",
                    "sourceIdentifier": "DEVOPS_WINE",
                    "gitCloneDepth": 1,
                    "type": "GITHUB",
                    "reportBuildStatus": false
                }
            ],
            "encryptionKey": "arn:aws:kms:us-east-1:111669150171:alias/aws/s3",
            "arn": "arn:aws:codebuild:us-east-1:111669150171:project/StorefrontApi",
            "secondarySourceVersions": [
                {
                    "sourceVersion": "refs/heads/staging",
                    "sourceIdentifier": "DEVOPS_WINE"
                }
            ]
        }
    ]
}

Answer 1

Apparently, at time of writing, CodeBuild does not use the native git client to fetch the source from GitHub. I understand that the CodeBuild internal teams have an internal feature request to move from whatever they're using to the native git client to improve performance.

Does your repository, by change, have lots of large files in its history? You can use this answer for a command to run to analyze your repository.

If you have lots of large files in your history and you're able to remove them, you can then use a tool like BFG Repo Cleaner to rewrite history. That should speed up the DOWNLOAD_SOURCE phase.

Also, if you have a dedicated support plan with AWS, you should reach out to your TAM to upvote the feature request to move to native git for GitHub source downloads.