'pyspark self class variable using to read csv file for dataframe
I have to call csv file to read data frame as below format, i can read normal file read but using self variable , i am not aware of it. kindly help us.
from pyspark.sql import SparkSession
class ChargePointsETLJob:
input_path = 'data/input/electric-chargepoints-2017.csv'
output_path = 'data/output/chargepoints-2017-analysis'
def __init__(self):
self.spark_session = (SparkSession.builder
.master("local[*]")
.appName("ElectricChargePointsETLJob")
.getOrCreate())
def extract(self):
pass
def transform(self, df):
pass
def load(self, df):
pass
def run(self):
self.load(self.transform(self.extract()))
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|