'signed integer is greater than maximum during zip file read boto3
Below is the requirement: Unzip files present in S3 folder to another S3 folder using AWS Databricks.
I have used the below code to unzip the files in the S3,for small files it works fine, although for zip files with the size of > 1.5 GB it doesn't working and throws below error.
OverflowError: signed integer is greater than maximum
Below is the code snipped iam using
s3_resource = boto3.resource('s3')
bucket = "bucket"
unzip_folder = 'unzipfolder/'
for key_file in zip_paths:
zip_obj = s3_resource.Object(bucket_name=bucket,
key=key_file)
buffer = io.BytesIO(zip_obj.get()["Body"].read())
z = zipfile.ZipFile(buffer)
for filename in z.namelist():
file_info = z.getinfo(filename)
s3_resource.meta.client.upload_fileobj(
z.open(filename),
Bucket=bucket,
Key= unzip_folder + f'{filename}'
)
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|


