Databricks

The Databricks plugin lets formae manage Databricks workspace resources: all-purpose clusters, instance pools, and workflow jobs.

Repository: formae-plugin-databricks

Installation

git clone https://github.com/platform-engineering-labs/formae-plugin-databricks.git
cd formae-plugin-databricks
make install

This builds the plugin and installs it to ~/.pel/formae/plugins/databricks/. The formae agent discovers installed plugins automatically on startup. Requires Go 1.26+.

Configuration

Target

Configure a Databricks target in your Forma file:

import "@formae/formae.pkl"
import "@databricks/databricks.pkl"

target: formae.Target = new formae.Target {
    label = "my-databricks"
    config = new databricks.Config {
        host = "https://your-workspace.cloud.databricks.com"
    }
}

Credentials

The plugin uses the Databricks SDK's default credential chain. Credentials are never stored in the target config. Pick whichever strategy fits your environment.

Databricks CLI (local dev):

export DATABRICKS_HOST="https://your-workspace.cloud.databricks.com"
databricks auth login --host "$DATABRICKS_HOST"

Personal access token:

export DATABRICKS_HOST="https://your-workspace.cloud.databricks.com"
export DATABRICKS_TOKEN="your-pat-token"

GitHub OIDC (CI/CD): The SDK's native github-oidc strategy exchanges a GitHub Actions OIDC token directly with Databricks, no long-lived secrets required. See the plugin repo's CI workflow for a working example.

Examples

Examples live in the plugin repository. Clone the repo and resolve Pkl dependencies before running:

git clone https://github.com/platform-engineering-labs/formae-plugin-databricks.git
cd formae-plugin-databricks
pkl project resolve examples/cluster-with-pool

Available examples:

Example	Description
cluster-with-pool	Instance pool, all-purpose cluster, and scheduled workflow job

# Evaluate an example
formae eval examples/cluster-with-pool/main.pkl

# Apply resources
formae apply --mode reconcile --watch examples/cluster-with-pool/main.pkl

Supported Resources

Type	Description	Native ID	Discoverable	Extractable
Databricks::Compute::Cluster	All-purpose compute clusters	clusterId	Yes	No
Databricks::Compute::InstancePool	Pools of pre-warmed instances for fast cluster startup	instancePoolId	Yes	No
Databricks::Jobs::Job	Workflow jobs with tasks and schedules	jobId	Yes	No

What's next

• Learn more about resolvables in Res
• See Target resolvables for cross-plugin target configuration

Release notes

See release notes for changes per version.