'How do dynamically pass options to SparkSession.read.csv in python
Client is requesting that I add functionality to pass option names and value from a **kwargs parameter into a call of SparkSession.read.csv() and I'm having trouble figuring out how to do that. The syntax for adding options is
SparkSession.read.option("option1", value).option("option2", value).csv()
or
SparkSession.read.options(option1=value, option2=value).csv()
In scala you can pass in a map which would make this easy but it appears you cannot do that in python. Is there a way that I can dynamically add these options? I know there are some workarounds for this but the client is specifically asking for this method.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|