'Adding a step in AWS EMR with --files option fails
I am adding a step into my EMR cluster using CLI as below:
aws emr add-steps --cluster-id j-258HYT78FYMWG --steps Type=Spark,ActionOnFailure=CONTINUE,Args=[--py-files,\\"s3://somebucket/app/jobs.zip,s3://somebucket/shared.zip,s3://somebucket/libs.zip\\",--files,s3://somebucket/config.json,s3://somebucket/main.py,--job,retail]
The expectation is that the config.json specified with --files flag will be copied into the working directory of the nodes and can be accessed in the application. But the application fails with the error as below when attempting to the read the config.json file:
FileNotFoundError: [Errno 2] No such file or directory: 'config.json'
Advises welcomed !
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
