How do i automate Databricks notebook from Azure Devops

Question

I am having all my codes in Azure Databricks notebook. I want to automate execution of my notebook using azure devops.

I have linked my Databricks Repos with Azure Repos. Created a build pipeline to create artifact of my code. In release pipeline.

I was able to deploy my code to lower environment(Here same Databricks workspace where I have my code). But I was not able to execute/Run notebook directly from Azure Devops.

How do I deploy code from my lower environment to higher environment(Prod Server) Databricks workspace.

To deploy code to higher environment Is it same as deploying to lower environment. Just changing details of workspace. Like Databricks Token, Workspace URL, etc.

Answer 1

You need to create release pipeline and push your code to GIT repo.
Follow this Microsoft documentation for details -
https://docs.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/ci-cd-azure-devops#:~:text=The%20release%20pipeline%20deploys%20the,Pipelines%20menu%20and%20click%20Releases.

Answer 2

This demo shows how to do that. The steps include:

• Testing
  • Update checkout at Databricks workspace - for example, you may have a separate checkout for testing. You can do that via Repos REST API, or via databricks-cli (databricks repos update command)
  • Triggering execution of tests by using the Nutter library
  • Reporting testing results to DevOps
• Release
  • If tests are successful, you can push updates to staging/production environment - this could be done via DevOps Release pipelines
  • You can use the same Repos REST API or databricks-cli to update checkout in staging/production