Category "luigi"

Airflow/Luigi for AWS EMR automatic cluster creation and pyspark deployment

I am new to airflow automation, i dont now if it is possible to do this with apache airflow(or luigi etc) or should i just make a long bash file to do this. I

Writing a Luigi Target as a csv with Pandas

I have a basic Luigi pipeline that I'm writing. The pipeline will download Apple stock data and create a CSV out of it. The following is what I've written: # Do