'Python Loop for AWS Data
I'm very new to python so this should be an easy solution for somebody more experienced.
I have a bunch of CSV files extracted from AWS. I need a loop to collate all these files into a single dataframe and add an extra column which has the Year and Month which can be retrieved from the object location string..
The CSV Files are all named the same with only the year and month changing. Currently the code I have is the following which extracts the data for 2021, month 02.
import boto3
import pandas as pd
from io import StringIO
client = boto3.client('s3',aws_access_key_id='XXXXXXX',aws_secret_access_key='XXXXXX')
try:
obj = client.get_object(Bucket='bucket1',Key='AccountFeed_V1/year=2021/month=02/data.csv')
body = obj['Body']
csv = body.read().decode('utf-8')
df = pd.read_csv(StringIO(csv))
print(df)
except Exception as e:
print(e)
Over to the gurus. Thanks!
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
