'Scheduling multiple spark streaming jobs and its dependency management
I have a requirement where I need to support adhoc runnable task which are spark streaming jobs. This is job scheduling mechanism should support orchestrating , monitoring , rescheduling and cancelling operations of these stream jobs. Error handling such as stream complete notifications , steam error notification is also one of the area that it is expected to support.
I tried to explore Airflow. Owning to limited knowledge on Airflow, I am not able to find anything that can help me to orchestra this sparked streaming jobs.Airflow seems supporting scheduling but with batch and intervals.
Could you please suggest any libraries or any mechanism using open-source technology that can help me achieve this particular requirement.References to enterprise solutions are also welcome.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|