'Airflow - Push xcom from SparkSubmitOperator
I need to push a value obtained inside of a spark job which is written in python. After this I will use that value to feed a Branch Operator and decide which path to take in the job flow.
So far, I have been trying to extend the SparkSubmitOperator to allow that, but with no success. I've found this and this, but none of those solutions seem to work for me, although they helped me understand a bit more how I might approach the problem.
Is it possible to modify de SparkSubmitOperator in a way I can use xcom_push from the python file running inside of it?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|