Cluster creation and management
1. Cluster creation and management
In this lesson, we will learn how to create, list, run and delete Databricks All-Purpose clusters.2. Serverless vs. managed infrastructure
The Databricks jobs you've seen so far can run on multiple types of infrastructure. Either clusters that we create and manage or serverless infrastructure. Running on serverless infrastructure means we don't have to concern ourselves with managing the specific servers that the code will run on, instead we can just focus on what our code will do and pay on-demand every time our code is ran. In contrast, we can create our own spark clusters that are customized to meet our specific use case. For long running jobs, such as training AI models or running data engineering pipelines, it can be more cost effective to create our own clusters instead of using serverless.3. Create a Databricks Spark cluster
The Databricks `clusters` service can be accessed via the `clusters` attribute of the `WorkspaceClient` object. We will use this service to create and manage clusters in our Databricks workspace. The `.create()` method of the Databricks `service` creates a new Spark cluster in our authenticated Databricks workspace. A Spark cluster is a group of servers that work together to process big data. There are many optional params that can be used to customize the new cluster to your needs. Here, we use the `cluster_name` param to pass in a name to identify the cluster being created. We set the `spark_version` of the cluster to be the latest stable version. We use `autotermination_minutes` to specify the number of minutes the cluster can be inactive before terminating to 20 minutes. And we use the `num_workers` param to specify that there should be 3 worker nodes available in this cluster. We use `.result()` to wait.4. Lists clusters
We can use the `list()` method of the `clusters` service to list all of the clusters in the authenticated Databricks workspace. It returns a list of `cluster` objects. In this example, we iterate through all of the `cluster` objects and print the `cluster_id` attribute of each cluster.5. Start a cluster
We can start a Databricks cluster by using the `start()` method of the `clusters` service and passing in the `cluster_id` of the cluster in your workspace that you want to start. In this lesson, we will assume that the `cluster_id` we are interested in is stored in an environment variable called "DATABRICKS_CLUSTER_ID".6. Check the state of a cluster
To retrieve information about a cluster, we can pass the `id` of the cluster to the `.get()` method in the `clusters` service. This function returns a dictionary with keys that contain information about the given cluster. We can use the `state` key to retrieve that state of a cluster, which informs us if it is in a Terminated, Idle or Running state.7. Delete a cluster
We can use the `.delete()` method of the `clusters` service to delete the Databricks Spark cluster with a specified `cluster_id`. This terminates the spark cluster with the specified ID and removes it asynchronously. Once the removal is complete, the cluster will still exist in the workspace, but it will be in a `TERMINATED` state.8. Let's practice!
Congratulations on learning how to create and manage Databricks Spark clusters. You can use this knowledge to create and use clusters customized to your specific use case.Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.