'How to access other Python files passed using spark-submit?

I have this codes.zip that are being passed to spark-submit

codes.zip
|-- src/
|  |-- conf/
|  |  |-- conf.yml
|  |-- main.py

In my main.py, I want to read the conf.yml but I don't get how paths inside a spark driver/executor work

main.py should look like this:

import yaml

with open("conf/conf.yml", "r") as f:
    config = yaml.full_load(f)

However, this is failing because the path is now different when passed to the Spark driver/executor I assume



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source