Flink can not send http request to S3 bucket on localstack

Question

I'm trying to call the getObject api to get an external file on S3 bucket0 within my Flink job, but it keeps getting a SdkException from my localstack setup:

 org.apache.flink.kinesis.shaded.com.amazonaws.SdkClientException:
 Unable to execute HTTP request: mybucketName.s3.localstack

This is how I created my resources with docker compose and localstack

container_name: localstack
image: localstack/localstack:0.12.15
ports:
  - "14566:4566"
expose:
  - "4566"
environment:
  - DEFAULT_REGION=us-east-1
  - AWS_ACCESS_KEY_ID=test
  - AWS_SECRET_ACCESS_KEY=test
  - EDGE_PORT=4566
  - SERVICES=s3
  - AWS_CBOR_DISABLE=1

I have tried to replace the endpoint with http://localstack:4566 and http://s3.localstack:4566, but I still see the error

import org.apache.flink.kinesis.shaded.com.amazonaws.services.s3.AmazonS3ClientBuilder;
    
AwsClientBuilder.EndpointConfiguration endpointConfiguration =
        new AwsClientBuilder.EndpointConfiguration(
            "http://s3.localstack:4566",   // also tried localstack:4566 and 127.0.0.1:4566 etc..
            region);

s3Client = AmazonS3ClientBuilder.standard()
        .withEndpointConfiguration(endpointConfiguration)
        .build()


S3Object s3Object = s3Client.getObject(bucketName, objectLocation);

Anyone knows why I'm getting this issue? Sorry the SdkException only shows unable to execute HTTP and does not output lots for context/info. Thank you.

Answer 1

I think I have found the solution, I should call .enablePathStyleAccess() in my client

    AmazonS3ClientBuilder.standard()
        .withEndpointConfiguration(endpointConfiguration)
        .enablePathStyleAccess()
        .build()

Please refer to this post