Inefficient Use of Interactive Clusters
Matt Weingarten
Service Category
Compute
Cloud Provider
Databricks
Service Name
Databricks Clusters
Inefficiency Type
Misconfiguration
Explanation

Interactive clusters are intended for development and ad-hoc analysis, remaining active until manually terminated. When used to run scheduled jobs or production workflows, they often stay idle between executions—leading to unnecessary infrastructure and DBU costs. Job clusters are designed for ephemeral, single-job execution and automatically terminate upon completion, reducing runtime and isolating workloads. Using interactive clusters for production jobs leads to cost inefficiencies and weaker workload boundaries.

Relevant Billing Model

Databricks compute costs consist of:

  • Databricks Units (DBUs): Billed per hour based on cluster type and instance class
  • Cloud Infrastructure Charges: Passed through from the underlying cloud provider, billed per second or minute

Interactive clusters accrue charges continuously while running, regardless of workload activity. Job clusters are provisioned on demand and shut down automatically, incurring charges only for the job’s runtime.

Detection
  • Identify scheduled jobs assigned to interactive clusters
  • Query system tables or logs to verify cluster assignment per job
  • Review cluster uptime relative to job duration to assess idle overhead
  • Evaluate whether workspace policies restrict interactive cluster use for jobs
  • Confirm with engineering teams if job clusters can be safely adopted
Remediation
  • Reassign scheduled jobs to ephemeral job clusters
  • Apply workspace policies to enforce job cluster usage for scheduled workflows
  • Educate users on the differences between cluster modes and their appropriate use cases