How can I add a final step to a Sagemaker Pipeline that runs even if other steps fail

Question

Is there a way to add an end step to a sagemaker pipeline that still runs at the very end (and runs code) even if other previous steps fail. Before I thought we could make it a Fail Step but that only lets you return an error message and doesn’t let you run code. If we made it a conditional step how would we make sure it ran at the very end without depending on any previous steps. I thought of adding all previous steps as a dependency so it runs at the end, but then the end step wouldn't run if any step before that failed.

I tried using the fail step, but I can't provide code. I tried putting it with dependencies but then it won't run if other steps fail before it. I tried putting no dependencies, but then it won't run at the end.

Answer 1

Pipelines doesn't currently have a finally construct like this. Any step failure will stop the pipeline immediately.

This might be added in the future, but the best way to accomplish this now would be an EventBridge rule on pipeline status change that triggers a Lambda, SageMaker Pipeline, etc which has your failure logic.