Model training with MLFlow in Databricks
1. Model training with MLFlow in Databricks
Hey DataCamp team! In this video, we will cover how to train your machine learning models efficiently using the Databricks Lakehouse Platform.2. Machine Learning Lifecycle
Let’s recall the steps in the machine learning lifecycle. So far, we have seen how Databricks provides functionality for the planning and preparation stage of the machine learning lifecycle.3. Model training and development
Now, we want to focus on how Databricks can help us with the next two stages of the machine learning lifecycle: model development and evaluation.4. Single-node vs. Multi-node
Before diving into Databricks-specific functionality, let's talk about scaling up machine learning workloads. Single-node machine learning is the classical approach for these workloads, as they will only use a single machine (for example, your laptop) to train a model. This is a great approach for experimenting and testing, as the setup is easy and quick. However, these models don't typically work well in a production setting, as they cannot scale how we need them to. The most popular example of these kinds of models are from the scikit-learn library. Most organizations are looking at moving into multi-node machine learning, which uses multiple compute resources (like a Databricks cluster) to train a model. These are much better for production workloads. While they require a slightly harder setup, they are much easier to maintain over time, as they are very scalable. This is where frameworks like Spark ML can be hugely helpful.5. AutoML
If you aren't very comfortable programming your own machine learning pipeline or just want a good foundation, you will benefit from the AutoML capability in Databricks. AutoML is a Glass Box approach to machine learning. Many other tools can automatically create a machine-learning model, but they don't provide much visibility into what is happening under the hood. With Databricks AutoML, the platform uses the most popular open-source libraries and can create a model that best predicts what you want it to predict based on your UI selections. To take it a step further, AutoML generates a notebook with the required code to reproduce, audit, or optimize the code for that model, so you know exactly what happened.6. MLFlow
Regardless of how you create and train your model, you will need a way to manage your work. Databricks hosts a managed version of MLFlow, an open-source framework that manages the end-to-end processes of machine learning development. With MLFlow, you can track, evaluate, manage, and deploy your models. Lucky for you, it even comes pre-installed on the Databricks ML Runtime! Here, we have some of the key components of MLFlow Tracking, which is the main component we need for our development. You can track your work in the context of a run, which incorporates all the steps you take to train a model. Most model frameworks allow you to auto-log everything about the model training with the autolog() method. Still, you can also explicitly log metrics or parameters that went into your model training.7. MLFlow Experiments
As you develop and train your models and track your work with MLFlow, you can review your work in MLFlow Experiments. These Experiments collect all the information across multiple runs into a single location, which Databricks manages. In these Experiments, you can review, sort, and compare the different runs you have completed as you train the model. You can compare these runs based on different metrics that are important to you. Once you have compared the runs, you can then find and promote the best model run to be the final version of that model.8. Let's practice!
Let's practice and review how Databricks makes the model training process easier!Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.