Administration

Cluster Configuration

Configure, size, and manage NATIS compute clusters for pipelines, notebooks, and SQL queries.

7 min read · Updated May 2025

On this page

Cluster Configuration Parameters
Cluster Policies

NATIS compute clusters are auto-provisioned on demand and automatically terminate after a configurable idle timeout. Admins set cluster policies that enforce size limits, auto-scaling bounds, and allowed instance types to control costs.

Cluster Configuration Parameters

Runtime Version — NATIS Runtime 13.x LTS (Spark 3.5, Python 3.10) recommended for production
Node Type — Standard (CPU) or GPU-enabled. Use r-series for memory-intensive ML workloads.
Autoscaling — Min/Max workers. NATIS scales between these bounds based on job parallelism.
Auto-Terminate — Cluster shuts down after N minutes of inactivity (default: 30 min)
Spot Instances — Enable to use preemptible spot VMs for 60-80% cost reduction (not for production streaming)
Init Scripts — Shell scripts run at cluster startup to install packages or configure environment

Cluster Policies

Cluster Policies constrain the options users can configure when creating clusters, ensuring compliance with cost and security standards. Admins create policies under Admin Console → Compute → Cluster Policies.

Always configure auto-termination for interactive clusters. Leaving clusters running idle overnight is the most common source of unexpected cost overruns.

JSON

{
  "spark_version": {
    "type": "allowlist",
    "values": ["13.3.x-scala2.12", "13.3.x-gpu-ml-scala2.12"],
    "defaultValue": "13.3.x-scala2.12"
  },
  "node_type_id": {
    "type": "allowlist",
    "values": ["Standard_DS3_v2", "Standard_DS4_v2", "Standard_NC6s_v3"],
    "defaultValue": "Standard_DS3_v2"
  },
  "autoscale.min_workers": {
    "type": "range",
    "minValue": 1,
    "maxValue": 4,
    "defaultValue": 1
  },
  "autoscale.max_workers": {
    "type": "range",
    "minValue": 2,
    "maxValue": 20,
    "defaultValue": 8
  },
  "autotermination_minutes": {
    "type": "fixed",
    "value": 60,
    "hidden": true
  },
  "enable_elastic_disk": {
    "type": "fixed",
    "value": true,
    "hidden": true
  }
}

Was this page helpful?

Thanks for your feedback!