'How to concurrently download multiple S3 objects using GO?
I am trying to download 4 objects from S3. Each of them is almost of size 1 MB. If I download them serially, I get a latency of around 40-50ms per object with a total latency of appx 200ms. However, If I try to download them concurrently, the total latency increases to 300ms. I am not sure why concurrent downloads are slower. Can someone help me with what is the correct way to download objects concurrently? Thanks!
Here is the GO code that I am using:
downloader := s3manager.NewDownloaderWithClient(client)
out := make([][]byte, len(keys))
wg := sync.WaitGroup{}
for i := 0; i < len(keys); i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
wb := &aws.WriteAtBuffer{}
_, err := downloader.Download(wb, &s3.GetObjectInput{
Bucket: aws.String(s3client.bucketName),
Key: aws.String(keys[i]),
})
out[i] = wb.Bytes()
}(i)
}
wg.Wait()
return out
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
