Databricks repos - unable to use dbutils.notebook.run with absolute path

Question

I'm unable to get an absolute path working with dbutils.notebook.run(). Using the absolute path with dbutils.fs.ls(with "file:/Workspace/Repos/user_email/Datalake/databricks/models/notebook") returns a FileInfo object. Running the same filepath string with dbutils.notebook.run(path) does start running a job, however with an error that the notebook was not found. The error includes a path which is the combination of the relative filepath (of the notebook from which I run the dbutils.notebook.run() command) concatenated with the path I to the referenced notebook.

Seems that dbutils.notebook.run is not able to access notebooks absolute with repos?

Thanks in advance.

Answer 1

/Workspace/Repos/user_email/... is the path on the driver machine, that is local to your cluster node. It works with dbutils.fs.ls because it works with files of different types. This path is a special replication of actual notebook/file that resides at path /Repos/user_email/... inside the workspace

On other hand, dbutils.notebook.run works with notebooks in the workspace, not with files, so you need either to specify that path as /Repos/user_email/..., but it's better to use relative paths, because it's more portable, for example, if someone else will use your code from the repository.