Evaluation: Metrics and plots in DVC

1. Evaluation: Metrics and plots in DVC

Welcome back! In this final video, we will learn about evaluating model performance with metrics and plots using DVC.

2. Metrics: changes in dvc.yaml

DVC provides a useful feature for monitoring model performance metrics, visualizing them through graphs, and comparing results across different experiments. This helps in choosing the best model for a specific machine-learning task. To monitor metrics, modify the dvc.yaml file by tracking the 'metrics.json' file under a 'metrics' key instead of using the 'outs' field. Since metrics files are typically small and text based, we set the 'cache 'parameter to false to track it using Git instead of DVC.

3. Printing DVC metrics

We can use the 'dvc metrics show' command to print current metrics on the terminal.

4. Compare metrics across runs

DVC provides the capability to contrast metrics across various experiments. Since we have committed the prior experiment in both DVC and Git, altering a hyperparameter and executing 'dvc repro' again would result in a new experiment. In this particular case, utilizing the 'dvc metrics diff' command to compare metrics reveals an enhancement in the performance of the model.

5. Plots: changes in dvc.yaml

Instead of tracking image files, DVC tracks the data files used to generate plots. The data file name serves as a plot output target reference under the plots key in the dvc.yaml file. Define a template for the plot style. Next, we organize the data by identifying column names from the data file and labeling the plot axes and title. Remember that setting cache to false means the plot file is tracked through Git.

6. Printing DVC plots to file

Similarly to metrics, we can now show plots by executing the 'dvc plots show' command in the terminal. This command requires a reference to the relevant target in the dvc.yaml file. By default, it yields a path to an HTML file that contains an interactive plot embedded within it.

7. Comparing DVC plots

To compare current state of plots with another branch, use the 'dvc plots diff' command. Specify the plot output target from the dvc.yaml and the branch we want to compare against. Alternatively, we can also specify the commit SHA. This facilitates the side-by-side visualization of plots and enables the comparison of changes between branches or commits.

8. Let's practice!

Now, let's check your understanding of managing metrics and plots in DVC.

Introduction to Data Versioning with DVC

IntermediateSkill Level

4.7+

72 reviews

This chapter provides a comprehensive introduction to Data Version Control (DVC), a tool essential for data versioning in machine learning. Learners will explore the motivation behind data versioning, understand its differences from code versioning, and experiment with a simple classification problem. They will review basic Git commands, learn about DVC, and practice setting up a repository. The chapter concludes with an overview of DVC’s features and use cases, including versioning data and models, CI/CD for machine learning, experiment tracking, pipelines, and more.

Exercise 1: Data Versioning Motivation Exercise 2: Anatomy of a Machine Learning Model Exercise 3: Differences Between Data and Code Versioning Exercise 4: Understanding Hyperparameters Exercise 5: Introduction to DVC Exercise 6: Working with Git CLI Exercise 7: Review DVC CLI Exercise 8: DVC features and use cases Exercise 9: DVC pipelines Exercise 10: CI/CD for machine learning

This chapter delves into the setup of DVC, encompassing aspects such as installation, initialization of the repository, and the utilization of the .dvcignore file. It further navigates through the exploration of DVC cache and staging files, imparting knowledge on how to add and remove files, manage caches, and comprehend the underlying mechanisms using the MD5 hash. The chapter also elucidates on DVC remotes, distinguishing them from Git remotes, and guides you on how to add, list, and modify them. Lastly, it teaches you how to interact with these remotes by pushing and pulling data, checking out specific versions, and fetching data to the cache.

Exercise 1: DVC Setup and Initialization Exercise 2: Setting up DVC Exercise 3: .dvcignore Patterns Exercise 4: DVC Cache and Staging Files Exercise 5: Working with DVC Cache Exercise 6: Understanding .dvc Files Exercise 7: Configuring DVC Remotes Exercise 8: Purpose of DVC Remotes Exercise 9: Setup a DVC Remote Exercise 10: Interacting with DVC Remotes Exercise 11: Versioning Data using DVC Remote Exercise 12: Checking out Versioned Data