How to improve performance of initial calls to AWS services from an AWS Lambda (Java)?

Question

I recently tried to analyze some performance issues on a service hosted in AWS Lambda. Breaking down the issue, I realized that it was only on the first calls on each container. When isolating the issue, I found myself creating a new test project to get a simple example.

Test project (You can clone it, build it mvn package, deploy it sls deploy and then test it via the AWS Management Console.)

This project has 2 AWS Lambda functions: source and target. The target function simply returns an empty json {}. The source function invokes the target function using the AWS Lambda SDK.

The approximate duration of the target function is 300-350 ms on cold starts and 1ms on hot invokes. The approximate duration of the source function is 6000-6300ms on cold starts and 280ms on hot invokes.

The 6 seconds overhead on the cold starts of the source function appear to be 3 seconds of getting the client and 3 seconds of invoking the other function, in hot invokes that is 3ms and 250ms respectively. I get similar times for other services like AWS SNS.

I don't really understand what it is doing in those 6 seconds and what I can do to avoid it. When doing warmup calls, I can get the client and store the reference to avoid the first few seconds, but the other few seconds come from actually using the other service (SNS, Lambda, etc), which I can't really do as a no-op.

So, do other people experience the same cold start durations and what can I do to increase the performance on that? (other than bringing the memory setting up)

Answer 1

Provisioned concurrency helps with The the code initialization duration you are having. Other than that, it targets to another overhead coming from execution environment setup for your function’s code.

Refer to Turning on Provisioned Concurrency section here.

Answer 2

aws-lightweight-client-java is a standalone jar (no dependencies) and is less than 60K. It was built with the exact purpose of reducing Java Lambda cold start times which it does considerably and it is easy to use (though you might have to check the AWS API docs for your task). I've found that with the AWS SDK S3 jar my cold start time was about 10s and with this lightweight client it's down to 4s (this is with 512MB memory allocated). Allocating 2GB memory to the Lambda yields a cold start time of 3.6s with the AWS SDK and down to 1s with the lightweight client.

The mere fact that the library is making https calls does bring about the loading of 2000 or so classes so it's hard to go a lot quicker than 1s (unless there's some cool https library out there that is much more efficient in this regard).

Answer 3

Basically, there are a set of recommendations that I am using as a cheat sheet every time I have to optimize lambda performance.

1. Use SDKv2. I have seen a lot of times when AWS SDKv1 and v2 are completely incompatible. Migration from v1 to v2 can be easy but sometimes API changes are so huge that you simply cannot find the corresponding method in V2. But if you can then you better do so. V2 introduces a lot of performance improvements so it’s a rule of thumb “If it’s possible then use V2 whenever you can”

2. Using defined credentials provider. AWS SDK has quite an interesting way of detecting the credentials. It passes multiple steps trying to figure out the proper credentials until it finds or fails to find any.

• Java system properties
• Environment variables
• Web identity token from AWS STS
• The shared credentials and config files
• Amazon ECS container credentials
• Amazon EC2 instance profile credentials

All these steps take time and you save some milliseconds by specifying exact credentials providers. Like this:

S3Client client = S3Client.builder()
       .credentialsProvider(EnvironmentVariableCredentialsProvider.create())
       .build();

This way SDK wouldn’t traverse all possible sources of creds and immediately detect the correct one.

3. Initialize everything prior to the execution Simple advice to follow. You may want to simplify things and put all the initialization into the handler method. Better don’t do this, but try to put as much initialization into the constructor as possible. It may reduce latency for repetitive lambda invocations.

4. Reduce jar size To reduce lambda cold start one of the not obvious advice is to reduce the jar size. Java developers usually don’t care to include a few more libraries to avoid reinventing the wheel. But in the case of lambda, you better take a closer look at your pom.xml and clean everything unnecessary. Cause a bigger jar means a longer cold start.

5. Avoid using any DI I don't think you wanted to use any kind of DI. But in case you do try to avoid it. Lambda's purpose is to be small and lightweight. And DI will dramatically increase cold start and it doesn't make a lot of sense to wire up 2-3 classes.

6. Use Tiered compilation Java Just in time compilation has such a cool feature as a tiered compilation introduced since Java 8 was released. The purpose of JIT is to run the code and eventually reach the native code performance. It cannot be done immediately. But running the code and analyzing the hot spots JIT eventually interprets code almost as good as native. This can be achieved by collecting the profiling information in the background. This makes sense with your monolith application running in a servlet container for ages. But short-lived Lambda cannot benefit from these optimisations and it’s better to turn it off completely. To do this put these env variables: how to put env vars

For better understanding, I would refer to oracle docs: https://docs.oracle.com/javacomponents/jrockit-hotspot/migration-guide/comp-opt.htm#JRHMG119

7. Specify Region and HttpClient explicitly By default, AWS SDK comes with 3 different HTTP libs supported which are apache, netty and the built-in JDK HTTP client. Apache and Netty have a lot of features that standard built-in doesn’t have but we have to reduce cold start so prefer using built-in and excluding two others to keep less dependency on the resulting jar.

<dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>software.amazon.awssdk</groupId>
                    <artifactId>netty-nio-client</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>software.amazon.awssdk</groupId>
                    <artifactId>apache-client</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

Almost the same situation is with the region. It took some time to figure out the region lambda is deployed and this time can be reduced by specifying the region explicitly. Overall the resulting configuration should look like:

       .region(Region.US_WEST_2)
       .httpClient(UrlConnectionHttpClient.builder().build())
       .build();

8. Use RDS Proxy to have connection pooling In case you are planning to use Lambda with RDS then this advice may help you as well otherwise skip it. In “normal” Java applications it is common to use a pool of connections to reuse existing ones and save some time on establishing new ones. RDS Proxy service is coming to the rescue when you are using Lambda.

9. Increase the memory allocated Simple yet powerful advice. It can be that your Lambda can run out of memory with the standard 128 Mb allocated. And it seems correct to increase memory in this case. What is hidden and not obvious is that increasing memory allocated gives your lambda more CPU available. So the combination of increased memory allocated and more virtual CPU power available of course decreases the execution time. Giving your lambda more memory and CPU means increased costs. But less execution time. Instead of guessing which combination is better, I propose to use this tool: https://github.com/alexcasalboni/aws-lambda-power-tuning

10. Use provisioned concurrency Other guys already mentioned this. Probably the most simple and easy way to solve the problem. But it incurs additional costs. Provisioned concurrency means AWS will keep execution context ready for you to be used thus decreasing lambda cold start. You can specify the number of provisioned instances and enjoy lambda being warmed up for you.

There is exotic advice to use Graal VM but I think my answer is long enough.

Answer 4

(im new to python so please dislike if not helpful) why don't you use '>'

'>' means that the value on the left is higher than the one on the right.

example

new_list = [] 
for item in list:
  if item > 0:
    new_list.append(item)