'elegant means of switching between S3 platforms with boto3
I have some code that uses boto3 and I also have the requirement to be able to switch between using AWS S3 and an S3 compatible on-premises storage platform, this is the code I use for connecting to the on-premises storage platform:
s3 = boto3.resource(service_name = 's3',
use_ssl = False,
aws_access_key_id = aws_key,
aws_secret_access_key = aws_secret,
endpoint_url = aws_endpoint)
The code for establishing a connection to AWS would look something like this:
s3 = boto3.resource(service_name = 's3',
aws_access_key_id = aws_key,
aws_secret_access_key = aws_secret,
region_name = aws_region)
I obtain the various parameters that I pass into boto3.resource from environment variables, I really want to avoid code that uses the logic of say:
if os.getenv("AWS_REGION") is not None:
s3 = boto3.resource(service_name = 's3',
use_ssl = False,
aws_access_key_id = aws_key,
aws_secret_access_key = aws_secret,
endpoint_url = aws_endpoint)
.
.
.
Is there an elegant way of switching between platforms with s3.resource or in fact anything in boto3 ?, would I be better off using boto3.session and having different .credentials profiles for AWS and my on-premises appliance ?
session = boto3.Session(profile_name='dev')
s3 = session.resource('s3')
Solution 1:[1]
As users of AWS S3 who are more advanced than I will probably already know, you can access AWS S3 via a URL endpoint, the naming convention of which is:
protocol://service-code.region-code.amazonaws.com
Therefore, is the region is us east 2, you would use the following:
http://s3.us-east-2.amazonaws.com
TL;DR there is no need to switch between using an AWS region and endpoint URL in your boto3 code, it will work equally well with a URL for an appliance/software defined storage platform or a URL for the relevant AWS region.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | ChrisAdkin |
