Kubeflow VS generic workflow orchestrator?

Question

i am struggling understanding the functional role of Kubeflow (KF) compared with other (generic) workflow orchestrator.

I know KF is oriented to ML tasks, and is built on top of Argo.

Two questions:

1. can KF be used at a higher level as a workflow orchestrator to perform more generic tasks (i.e. ETL) whose outcome might be useful in the following ML tasks?
2. can use all funcionalities of Argo within KF.
3. what can a generic workflow orchestrator (as Airflow, argo, etc.) do that KF cannot?

Answer 1

1. Yes, you can create a python function/ general containers with code baked in which executes whatever task you like.

• pre-defined component - https://www.kubeflow.org/docs/components/pipelines/sdk-v2/component-development/

• python component - https://www.kubeflow.org/docs/components/pipelines/sdk-v2/python-function-components/

2. KFP is an abstraction op top of Argo workflows. it gives you the ability to create Workflows using python instead of writing YAML files. Check out this article : https://towardsdatascience.com/build-your-data-pipeline-on-kubernetes-using-kubeflow-pipelines-sdk-and-argo-eef69a80237c

• since Argo Workflows development is advancing independently from KFP it's safe to assume there will be missing features in KFP (Which are the community will add according to demands).

3. that's a big question. in general, airflow has sensors, SLA feature/ huge store of operators/sensors/reports/plugins and a bigger community since it's not ML oriented.