kubeflow OutputPath/InputPath question when writing/reading multiple files

Question

I've a data-fetch stage where I get multiple DFs and serialize those. I'm currently treating OutputPath as directory - create it if it doesn't exist etc. and then serialize all the DFs in that path with different names for each DF.

In a subsequent pipeline stage (say, predict) I need to retrieve all those through InputPath.

Now, from the documentation it seems InputPath/OutputPath as file. Does kubeflow as any limitation if I use it as directory?

Answer 1

I have experimented with a trivial pipeline - no issue is observed if InputPath/OutputPath is used as directory.