Databricks Driver Documentation

This documentation explains the Databricks driver implementation and how connection parameters map to UI fields in the Gigantic application. The driver connects to a Databricks SQL Warehouse (or an all-purpose cluster) and works with the tables of one catalog and schema.

Databricks is listed under Cloud datasources in the driver selector and requires a Cloud license tier.

Connection Parameters

Notes:

• The form has no Port field. The driver connects over HTTPS to the host on port 443, or on the local port of the SSH tunnel when the tunnel is enabled.
• The User field of the "User / Password" authentication is ignored. Only the Password field is used, and it must contain the access token.
• Tables are always addressed as <catalog>.<schema>.<table>.

If any of the required values is missing the connection fails with one of these errors:

Databricks host is required
Databricks HTTP path is required
Databricks access token is required

Technical Details

The Databricks driver uses the official @databricks/sql Node.js library (DBSQLClient).

Connection process:

1. Opens a client with host, HTTP path and token
2. Opens a session with the catalog and schema as initial catalog and schema
3. Runs SELECT 1 to check the connection

Schema discovery:

• Lists the tables of the schema from <catalog>.INFORMATION_SCHEMA.TABLES (table types TABLE, MANAGED, EXTERNAL and VIEW)
• Reads the columns from INFORMATION_SCHEMA.COLUMNS
• Gets the row count with count(*) and the size from DESCRIBE DETAIL (sizeInBytes)
• Reads primary keys from TABLE_CONSTRAINTS and CONSTRAINT_COLUMN_USAGE. This only works with Unity Catalog; without it the tables have no primary keys
• Stores the table DDL from SHOW CREATE TABLE
• Inspects up to 4 tables at the same time

Type mapping:

Reading process:

1. Builds a SELECT with the rule where, limit and offset
2. Fetches rows in batches of the read batch size of the job or rule (when none is set, the driver batchSize, 10000) and streams them

Writing process:

1. Inserts use multi-row INSERT ... VALUES statements in batches of the write batch size of the job or rule (when none is set, the driver batchSize, 10000)
2. Only the columns that exist in the target table are written
3. Strings are converted to numbers for numeric columns; values that cannot be converted are written as NULL
4. Update and merge use MERGE INTO on the primary key columns. They fail if the table has no primary key
5. When Retry On Fail is enabled, failed batches are split in half and retried (see Drivers Overview)

Write modes:

When the target table does not exist, it is created from the source DDL.

Authentication Options

SSH Tunnel Support

Enable Enable SSH? to reach the workspace through an SSH server. The driver then connects over HTTPS to the local port of the tunnel, which forwards the connection to the host on port 443.

API Endpoints Used

Databricks connections are used in:

• Tap creation (data source discovery)
• Sink creation (data destination for anonymized data)
• Pipeline execution (data extraction and loading)

Node.js Driver Dependency

• Library: @databricks/sql
• Purpose: Databricks SQL Warehouse and cluster connectivity

Custom Params

Custom Params (YAML) are merged into the driver configuration, so they can set or override these keys:

httpPath: /sql/1.0/warehouses/abc123   # Overrides the HTTP Path field
token: dapi0123456789abcdef            # Access token, overrides the Password field
catalog: main                          # Overrides the Catalog field
batchSize: 5000                        # Read/write batch size when the job or rule sets none (10000)

Example Configurations

Unity Catalog table in a custom schema

catalog: sales

With the Schema field set to crm, tables are read as sales.crm.<table>.