Why ML requires high-quality data
1. Why ML requires high-quality data
Data is used by machine learning models to derive predictive insights and make repeated decisions. However, the accuracy of those predictions relies on large volumes of data that is correct and free of errors. Data is considered low quality if it's not aligned to the problem, or is biased in some way. If you feed an ML model low quality data, it's like teaching a child with incorrect information. An ML model can't make accurate predictions by learning from incorrect data. So, how can you ensure that you have quality data when training an ML model? To assess it's quality, data is evaluated against six dimensions. Completeness, uniqueness, timeliness, validity, accuracy, and consistency. Let's explore what each of these mean in more detail. The completeness of data refers to whether all the required information is present. If the data is incomplete, then the model will not learn all the patterns that are necessary to make accurate predictions. Take, for example, the training of an ML model that's reliant on a data set of customer transactions. If some transactions are missing critical information, such as the date of the transaction, the accurate training of the model will be affected. Data should be unique. If a model is trained on a data set with a high number of duplicates, the ML model may not be able to learn accurately. This is because it will be confused by the duplicate records and won't be able to accurately identify patterns. For example, if you're training a model to identify a breed of dog from a photo, it's important to have images of many different unique breeds. If the data set contains many thousands of images, but most of them are just photos of Labradors, the model will find it difficult to correctly identify most other breeds accurately. The timeliness of the data refers to whether the data is up-to-date and reflects the current state of the phenomenon that's being modeled. If the data is not timely, then the model might be making predictions based on outdated or irrelevant information. Training an ML model to predict stock market fluctuations might rely on a data set of stock prices. If the data is several months old, it's untimely for making current predictions. Validity means the data conforms to a set of predefined standards and definitions, such as type and format. Validity also ensures that data is in an acceptable range. An example of invalid data is a date of 08-12-2019, when the standard format is defined as year, month, and date. Accuracy reflects the correctness of the data, such as the correct birth date or the accurate number of units sold. For example, in a data set of images, some images might be labeled as dogs when they actually show cats. Note how accuracy is different from validity. Whereas validity focuses on type, format, and range, accuracy is focused on form and content. Finally, the consistency of the data refers to whether the data is uniform and doesn't contain any contradictory information. If data is inconsistent, then an ML model might be unable to make accurate predictions. If the same entity appears with different names or values across different parts of the data, it would lead to inconsistent data. For example, in a dataset of customer information, the same customer might appear as John Smith in one place, and J.Smith in another. Remember, data is the only lens through which a model views the world. Anything the model can't see, it assumes doesn't exist. So it's your responsibility to provide the model with complete and correct data. The good news is that most of these problems can be solved simply by getting more high quality data, but you have to be purposeful in collecting that data.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.