'How do I pass a non iterable param in a pool.starmap call?
I have the below code. I'm trying to figure out how to pass the s3_resource param in the pool.starmap call to copy_s3_to_s3. I've tried several ways and keep getting errors. I can't use a global for this, I have to pass it in.
def copy_s3_to_s3(s3_resource, from_bucket, from_key, to_bucket, to_key):
copy_source = {'Bucket': from_bucket, 'Key': from_key}
s3_resource.meta.client.copy(copy_source, to_bucket, to_key)
def upload_multiprocess(s3_resource, from_bucket, keys_list_from, to_bucket, keys_list_to, num_proc=16):
print("number of proccesses in pool: ", str(num_proc))
with Pool(num_proc) as pool:
pool.starmap(copy_s3_to_s3, zip(repeat(from_bucket), keys_list_from, repeat(to_bucket), keys_list_to), 15)
pool.close()
pool.join()
return len(keys_list_to)
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
