1. Machine learning risks
Now that we know about how to assess model performance, here we will learn about the risks and how to identify them!
2. Poor performance
First one is performance. If models have poor performance metrics, this is an issue and you are not likely to have an effect on your business since the predictions are not accurate enough. One point to stress out though - make sure you are presented model performance on the test dataset, not training.
3. Low precision
So, in classification you might get low precision which means there are lots of false positives.
For example we might see a lot of customers incorrectly predicted as likely to buy something, while only 10% of those predictions were accurate.
In this case, we should assess what is the cost of a business action targeting a false positive prediction.
4. Low recall
Or, you might get low recall metrics when only a small fraction of the total number of observations of interest are captured by the model.
For example we might have a situation where the model only captures 25% of total fraudulent transactions, which means we will miss 75% of the actual fraudulent transactions.
In this case we need to improve the model or introduce other inspection mechanisms to capture more fraud.
5. Large error
In regression, you might get a large error when comparing the predicted values to the actual ones and take the average.
For example, what if your prediction on average has an error of 3.5 in predicting customer satisfaction rating on a scale of 5, or a 70% percentage error. It's pretty large, isn't it?
In regression, it's important to assess what is the cost of the mistake and what levels of error the business is willing to tolerate.
6. Non-actionable model use cases
Now, as discussed before, even the best models with best performance metrics can be unusable. It can happen anywhere - churn prediction, purchase prediction, machine failure prediction or fraud prediction.
The key question is how to build and test them correctly?
When the model is built, it's important to test it and assess if it can affect business outcomes. Can you reduce churn by incentivizing the customers who are likely to churn, or will you capture more fraudulent transactions if you introduce a process to manually review the ones that have been predicted as very risky?
7. A/B testing
So how do you test the model?
Once the model is built and performance reaches a desired level, it's time for an experiment or an AB test. Here, we will use churn prevention example.
We use the model to predict which customers are likely to churn. Then, we randomly split them into 2 groups for an AB test. One group predicted as at risk will receive specific retention campaigns, and the other one nothing - it will be our control group with no extra effort. Once the test runs, we compare the churn levels between them and see if the group A that has been targeted, has lower churn levels after the campaign than the control group. You might want to run the experiment a few times. If there are no lifts, then the model might not be actionable. If it is, potentially that's a great candidate for production in key business systems like CRM, website, app and so on.
8. What if tests don't work?
Now, if the tests don't give positive results, there are a number of things you can do.
First, getting more data - it's important for the business to work closely with machine learning teams to review which data is being used and what other data should be used to improve predictions. Business has a lot of experience and judgement of what is predictive and what isn't, and that information should be utilized.
Then, as discussed in the beginning, you should use that data to build causal models in order to understand what the drivers of the outcome are.
Another alternative is qualitative research and surveys, if you are trying to get to the reasons why customers are doing certain things.
Finally, changing the scope of the problem - narrow it down to be more specific, widen it to cover a broader question, or change the question altogether.
9. Let's practice!
Great, let's now go do some exercises!