'How to access other Python files passed using spark-submit?
I have this codes.zip that are being passed to spark-submit
codes.zip
|-- src/
| |-- conf/
| | |-- conf.yml
| |-- main.py
In my main.py, I want to read the conf.yml but I don't get how paths inside a spark driver/executor work
main.py should look like this:
import yaml
with open("conf/conf.yml", "r") as f:
config = yaml.full_load(f)
However, this is failing because the path is now different when passed to the Spark driver/executor I assume
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
