'Can not connect dbt cloud or dbt core to databricks

I am having issue connecting my dbt cloud and dbt core to databricks

I have read these 4 links, but still can not connect

https://docs.databricks.com/integrations/prep/dbt-cloud.html#connect-to-dbt-cloud&language-Cluster

https://docs.databricks.com/integrations/prep/dbt.html

https://docs.getdbt.com/reference/warehouse-profiles/databricks-profile

https://github.com/databricks/dbt-databricks

on dbt-cloud:

When I test the connection during the project creation step, it passed the test. however when I use the connection to create a job and run, it returns this message:"Cannot set database in spark!"

Edit: the issue once has been fixed but it comes back again.

original fix: The dbt-core connection issue has been fixed. The issue is caused by the python certificate issue on MacOS. please refer to this link for the solution

on dbt-core: this how I set up my profiles.yaml file base on the documentations:

    databrick_dbt_lakehouse:
      outputs:
        dev:
          host: adb-755xxxxxxx7.7.azuredatabricks.net
          http_path: /sql/protocolv1/o/755xxxx7/0517-xxxxxx-xxxxxx
          schema: default
          threads: 1
          token: dapi<my token>
          type: databricks
      target: dev

Note:

  1. for http_path i have tried both with or without backslash (/) before sql/...
  2. I assume the schema means database name. I have tried 2, but none of them works
  3. I use pipenv with python version 3.8.8

when I run dbt debug I got this message:

check failed:
   dbt was unable to connect to the specified database.
   The database returned the following error:

Runtime Error
   Database Error
   failed to connect

Please help, thanks

databricksazure-databricksdbtdatabricks-connectjava


Solution 1:[1]

Your http_path seems wrong. Here is an extract from my profiles.yml

databricks_dbt_demo:
  outputs:
    dev:
      host: adb-redacted.azuredatabricks.net

      http_path: /sql/1.0/endpoints/redacted  # for SQL Endpoint
      # http_path: /sql/protocolv1/o/{ORG-ID}/{CLUSTER-ID}  # for All-Purpose Cluster
      schema: your_database_here
      token: your_personal_access_token_here
      type: databricks
  target: dev

Are you using a cluster (e.g. All Purpose Cluster) or a SQL Endpoint?

Solution 2:[2]

Edit: This issue is came back again. I am writing this comment on the 3rd day after the original fix

When I run dbt run or dbt snapshot the returns this error message:

Encountered an error:
Runtime Error
   Database Error
      failed to connect

original fix:

The issue is caused by the python certificate issue on MacOS. please refer to this link for the solution.

The dbt cloud issue is caused by the incorrect schema name in the yaml file

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Anders Swanson
Solution 2

Related Questions