Create a cluster

Access the cluster creation interface

To create a cluster using the user interface, you must be in the Data Science & Engineering or Machine Learning persona-based environment. Use thepersona switcherif necessary.

Then you can either:

• ClickComputein the sidebar and thenCreate computeon the Compute page.

• ClickNew>Clusterin the sidebar.

Cluster policy

Cluster policiesare a set of rules used to limit the configuration options available to users when they create a cluster. Cluster policies have ACLs that regulate which specific users and groups have access to certain policies.

By default, all users have access to thePersonal Computepolicy, allowing them to create single-machine compute resources. If you don’t see the Personal Compute policy as an option when you create a cluster, then you haven’t been given access to the policy. Contact your administrator to request access to the Personal Compute policy or an appropriate equivalent policy.

To configure a cluster according to a policy, select a cluster policy from the政策dropdown.

What is cluster access mode?

Cluster access mode is a security feature that determines who can use a cluster and what data they can access via the cluster. When you create any cluster in Databricks, you must select an access mode.

You can upgrade an existing cluster to meet the requirements of Unity Catalog by setting its cluster access mode toSingle UserorShared. There are additional access mode limitations for Structured Streaming on Unity Catalog, seeStructured Streaming support.

Databricks Runtime version

Databricks Runtime is the set of core components that run on your clusters. All Databricks Runtime versions include Apache Spark and add components and updates that improve usability, performance, and security. For details, seeDatabricks runtimes.

You select the cluster’s runtime and version using theDatabricks Runtime Versiondropdown when you create or edit a cluster.

Cluster node type

A cluster consists of one driver node and zero or more worker nodes. You can pick separate cloud provider instance types for the driver and worker nodes, although by default the driver node uses the same instance type as the worker node. Different families of instance types fit different use cases, such as memory-intensive or compute-intensive workloads.

Cluster size and autoscaling

When you create a Databricks cluster, you can either provide a fixed number of workers for the cluster or provide a minimum and maximum number of workers for the cluster.

When you provide a fixed size cluster, Databricks ensures that your cluster has the specified number of workers. When you provide a range for the number of workers, Databricks chooses the appropriate number of workers required to run your job. This is referred to asautoscaling.

自動定量,磚動態ally reallocates workers to account for the characteristics of your job. Certain parts of your pipeline may be more computationally demanding than others, and Databricks automatically adds additional workers during these phases of your job (and removes them when they’re no longer needed).

Autoscaling makes it easier to achieve high cluster utilization, because you don’t need to provision the cluster to match a workload. This applies especially to workloads whose requirements change over time (like exploring a dataset during the course of a day), but it can also apply to a one-time shorter workload whose provisioning requirements are unknown. Autoscaling thus offers two advantages:

• Workloads can run faster compared to a constant-sized under-provisioned cluster.

• 自動定量集群可以減少整體costs compared to a statically-sized cluster.

Depending on the constant size of the cluster and the workload, autoscaling gives you one or both of these benefits at the same time. The cluster size can go below the minimum number of workers selected when the cloud provider terminates instances. In this case, Databricks continuously retries to re-provision instances in order to maintain the minimum number of workers.

Autoscaling local storage

If you don’t want to allocate a fixed number of EBS volumes at cluster creation time, use autoscaling local storage. With autoscaling local storage, Databricks monitors the amount of free disk space available on your cluster’s Spark workers. If a worker begins to run too low on disk, Databricks automatically attaches a new EBS volume to the worker before it runs out of disk space. EBS volumes are attached up to a limit of 5 TB of total disk space per instance (including the instance’s local storage).

To configure autoscaling storage, selectEnable autoscaling local storage.

The EBS volumes attached to an instance are detached only when the instance is returned to AWS. That is, EBS volumes are never detached from an instance as long as it is part of a running cluster. To scale down EBS usage, Databricks recommends using this feature in a cluster configured withAWS Graviton instance typesorAutomatic termination.

Local disk encryption

Some instance types you use to run clusters may have locally attached disks. Databricks may store shuffle data or ephemeral data on these locally attached disks. To ensure that all data at rest is encrypted for all storage types, including shuffle data that is stored temporarily on your cluster’s local disks, you can enable local disk encryption.

When local disk encryption is enabled, Databricks generates an encryption key locally that is unique to each cluster node and is used to encrypt all data stored on local disks. The scope of the key is local to each cluster node and is destroyed along with the cluster node itself. During its lifetime, the key resides in memory for encryption and decryption and is stored encrypted on the disk.

To enable local disk encryption, you must use theClusters API 2.0. During cluster creation or edit, set:

{"enable_local_disk_encryption":true}

SeeCreateandEditin the Clusters API reference for examples of how to invoke these APIs.

Here is an example of a cluster create call that enables local disk encryption:

{"cluster_name":"my-cluster","spark_version":"7.3.x-scala2.12","node_type_id":"r3.xlarge","enable_local_disk_encryption":true,"spark_conf":{"spark.speculation":true},“num_workers”:25}

Cluster tags

Cluster tags allow you to easily monitor the cost of cloud resources used by various groups in your organization. You can specify tags as key-value pairs when you create a cluster, and Databricks applies these tags to cloud resources like VMs and disk volumes, as well asDBU usage reports.

For clusters launched from pools, the custom cluster tags are only applied to DBU usage reports and do not propagate to cloud resources.

For detailed information about how pool and cluster tag types work together, seeMonitor usage using cluster and pool tags.

To configure cluster tags:

1. In theTagssection, add a key-value pair for each custom tag.

2. ClickAdd.

AWS configurations

When you configure a cluster’s AWS instance you can choose the availability zone, the max spot price, and EBS volume type. These settings are under theAdvanced Optionstoggle in theInstancestab.

Spark configuration

微調刺激就業,你可以提供自定義的Spark configuration propertiesin a cluster configuration.

1. On the cluster configuration page, click theAdvanced Optionstoggle.

2. Click theSparktab.

   

   InSpark config, enter the configuration properties as one key-value pair per line.

When you configure a cluster using theClusters API 2.0, set Spark properties in thespark_conffield in theCreate cluster requestorEdit cluster request.

To set Spark properties for all clusters, create aglobal init script:

dbutils.fs.put("dbfs:/databricks/init/set_spark_params.sh","""|#!/bin/bash||cat << 'EOF' > /databricks/driver/conf/00-custom-spark-driver-defaults.conf|[driver] {| "spark.sql.sources.partitionOverwriteMode" = "DYNAMIC"|}|EOF""".stripMargin,true)

Retrieve a Spark configuration property from a secret

Databricks recommends storing sensitive information, such as passwords, in asecretinstead of plaintext. To reference a secret in the Spark configuration, use the following syntax:

spark. {{secrets//}}

For example, to set a Spark configuration property calledpasswordto the value of the secret stored insecrets/acme_app/password:

spark.password {{secrets/acme-app/password}}

For more information, seeSyntax for referencing secrets in a Spark configuration property or environment variable.

Environment variables

You can configure custom environment variables that you can access frominit scriptsrunning on a cluster. Databricks also provides predefinedenvironment variablesthat you can use in init scripts. You cannot override these predefined environment variables.

1. On the cluster configuration page, click theAdvanced Optionstoggle.

2. Click theSparktab.

3. Set the environment variables in theEnvironment Variablesfield.

   

You can also set environment variables using thespark_env_varsfield in theCreate cluster requestorEdit cluster requestClusters API endpoints.

Cluster log delivery

When you create a cluster, you can specify a location to deliver the logs for the Spark driver node, worker nodes, and events. Logs are delivered every five minutes to your chosen destination. When a cluster is terminated, Databricks guarantees to deliver all logs generated up until the cluster was terminated.

The destination of the logs depends on the cluster ID. If the specified destination isdbfs:/cluster-log-delivery, cluster logs for0630-191345-leap375are delivered todbfs:/cluster-log-delivery/0630-191345-leap375.

To configure the log delivery location:

1. On the cluster configuration page, click theAdvanced Optionstoggle.

2. Click theLoggingtab.

   

3. Select a destination type.

4. Enter the cluster log path.

{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:ListBucket"],"Resource":["arn:aws:s3:::"]},{"Effect":"Allow","Action":["s3:PutObject","s3:PutObjectAcl","s3:GetObject","s3:DeleteObject"],"Resource":["arn:aws:s3:::/*"]}]}