'How to assume a AWS role in pyspark
I am currently using spark 3.1, and I am using
spark_context._jsc.hadoopConfiguration().set("fs.s3a.access.key", config.access_id)
spark_context._jsc.hadoopConfiguration().set("fs.s3a.secret.key", config.access_key)
to run read data from s3.
I have 2 questions here:
i) Which is the best way to run a pyspark job, AWS access_id and key or assume role and run it and why?
ii) Can someone please give an example of using role in pyspark job?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|