'glue to connect localstack s3
I'm creating glue for local development following the concept mentioned here.
I'm using localstack along with glue spark using Jupyter lab
As you can see below the screenprint, from my spark jupyter lab, I can able to connect to localstack s3 it's bucket and contents.
Jupyter lab is created usingdocker run -it -v ${PWD}/local_path_to_workspace/:/home/glue_user/workspace/jupyter_workspace/ -e AWS_ACCESS_KEY_ID=dummyaccess -e AWS_SECRET_ACCESS_KEY=dummysecret -e AWS_REGION=eu-west-1 -e DISABLE_SSL=true -e ENDPOINT_URL="http://localstack-glue:4566" --rm -p 4040:4040 -p 18080:18080 -p 8998:8998 -p 8888:8888 --name glue_jupyter_lab amazon/aws-glue-libs:glue_libs_3.0.0_image_01 /home/glue_user/jupyter/jupyter_start.sh to start the jupyter lab.
however, if I use below commands to access my localstack s3 bucket inside jupyter lab:
from pyspark import SparkContext
from awsglue.context import GlueContext
glueContext = GlueContext(SparkContext.getOrCreate())
inputDF = glueContext.create_dynamic_frame_from_options(connection_type = "s3", connection_options = {"paths": ["s3://glue-localstack-bucket-person/persons.json"]}, format = "json")
I'm getting access denied error.
An error occurred while calling o73.getDynamicFrame.
: java.nio.file.AccessDeniedException: s3://glue-localstack-bucket-person/persons.json: getFileStatus on s3://glue-localstack-bucket-person/persons.json: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: WGKYX9DAMXQF1P20; S3 Extended Request ID: Kvpk7Ri77fkrtmzeTCZv+VGoD1dfHUgbXdihVfXxTWlXLIb/vfU+y11jZPiFMGol2F+6sXqGUmw=; Proxy: null), S3 Extended Request ID: Kvpk7Ri77fkrtmzeTCZv+VGoD1dfHUgbXdihVfXxTWlXLIb/vfU+y11jZPiFMGol2F+6sXqGUmw=:403 Forbidden
So my question is how can I change session configuration to refer local stack endpoint with dummy access key and secret key?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
