Kubeflow Pipelines error on GCP - Run doesn't end

Question

After submitting the "run" using Jupyter notebook, when I go to the Kubeflow pipeline dashboard, I can see my "run" submitted & running but it doesn't end even after few hours & doesn't show any logs either.

I have tried running the "DEMO XG-BOOST" but the same issue persists & the "run" doesn't end.

Can someone please help me understand if there is any issue with the account settings or any other issue which I'm missing because I have tried to check documentation & other websites but couldn't understand why this occurs.

Please note that this "run" was working fine (creating the pipeline flow) on coursera using quicklabs (Covertype classifier)

error snapshot

Using below versions: kfp 1.8.12 ; kfp-pipeline-spec 0.1.14; kfp-server-api 1.8.1; Python 3.7.12

If there is any other information which I have missed to put above, please let me know so I could share that information that could help us solve this issue.

Answer 1

The issue mentioned above has been resolved. The part where "DEMO XG-BOOST" run doesn't end has been resolved by selecting "Use emissary executor" option while creating the pipeline.

see the snapshot

When we launch the pipeline with this setting, it solved the issue & we can run complete pipelines now.

More details: We took support from GCP & they have mentioned that the issue might have been caused by a recent upgrade of the GKE cluster which removes the docker runtime (https://www.kubeflow.org/docs/components/pipelines/installation/choose-executor/#docker-executor). Namely, the Docker executor is the default workflow executor and depends on docker container runtime, which is deprecated on Kubernetes 1.20+. We were using a GKE cluster whose version was 1.21.6. Hence the issue. So, we used the documentation (https://www.kubeflow.org/docs/components/pipelines/installation/choose-executor/#migrate-to-emissary-executor) & migrated to the Emissary executor (instead of Docker) which has solved our issue.