1. Introducing full automation and best practices to MLOps
Great work so far, and welcome to this lesson on automation and best practices in MLOps.
2. The MLOps lifecycle
Activities in the design phase usually require intervention from business stakeholders and domain experts. Even though the design of ML systems cannot be automated, (yet), we can apply processes with reproducible steps to streamline this phase.
3. Maturity levels in MLOps
The level of automation in an ML system determines its maturity. We distinguish between three levels of automation: manual, semi-automated, and fully automated.
4. Manual ML workflow - Ad hoc experimentation
Most teams begin their ML journey with a manual workflow, where a typical ML pipeline is used to run adhoc experiments without tracking or logging, making reproducibility difficult. After training, models are handed over from Development to Operations for deployment.
5. Semi-automated ML workflow
Semi-automated workflows improve the data scientist's experimentation process by orchestrating the steps of their ML experiment, making it faster. This orchestrated experiment is the starting point for a semi-automated workflow, which can be further automated with MLOps orchestration for efficient management of machine learning pipelines.
6. Semi-automated ML workflow
In addition, we use a source code repository: an archive of the code being worked on. We introduced also a new component in the architecture: the feature store used to feed data to our experiment in a reusable way.
7. Semi-automated ML workflow
On the Operations side, an automated ML pipeline is executed once new code is delivered via the source code repository. The metadata produced by the execution of the automated pipeline is saved in the metadata store.
8. Semi-automated ML workflow
An automated pipeline produces a trained model, which can be stored in a model registry. Continuous Delivery automates this process by using the latest trained model to deliver updated prediction services.
9. Semi-automated ML workflow
Predictions generated by the prediction service are monitored continuously, allowing the system to trigger the automated pipeline automatically. The ML team can also use performance monitoring results for ad-hoc experimentation.
10. Fully automated ML workflow
A fully automated MLOps system elevates the semi-automated architecture by incorporating the deployment of the automated ML pipeline into a Continuous Delivery process.
This results in a fully automated MLOps system where new pipelines are continuously delivered, deploying updated prediction services using the latest trained ML models.
With this level of automation, organizations can enjoy a streamlined and efficient ML workflow, enabling faster and more reliable deployment of machine learning models.
11. Automation in the ML life cycle - Design
The design phase, which involves input from business stakeholders and domain experts, cannot be fully automated. However, using processes with reproducible steps (such as the Team Data Science Process, a framework for structuring and standardizing the development of data science projects) and documenting work and results thoroughly can streamline this phase. Reproducibility and documentation are crucial throughout the ML lifecycle.
Documentation can include information about the project's problem definition, and success metrics.
12. Automation in the ML life cycle - Development
While manual effort is necessary in the ML development phase, automating tasks can improve the process. Best practices include writing clean code, using version control, and orchestration tools for automatable tasks like data preprocessing, model selection, hyperparameter tuning, and more. Although the entire workflow may not be fully automated, automating these tasks can significantly improve efficiency and accuracy.
13. Automation in the ML life cycle - Operations
The operations stage is where automation shines.
As we will see later, it is important to use automated testing as part of our workflows. Automated testing of our workflows entails more than traditional testing since we must also ensure that data and models are tested.
Soon, we will also learn about Continuous Integration, Continuous Deployment, Continuous Training, and Continuous Monitoring.
14. Let's practice!
Now, let's practice!