1. Communication management
Great work on identifying machine learning mistakes! Now we'll look into some of the best practices of managing communication between business and machine learning teams.
2. Working groups
Once the project has been funded and initiated, it is important for the business to dedicate a point of contact and have a recurring progress meeting to define and track the following topics. You should define the frequency internally but it shouldn't be more frequent than weekly while not less than monthly.
In those meetings we should define the business requirements, revue the model and products, identify inference and prediction use cases, baseline model results, outline market testing strategy, and talk about potential production systems to use this model in.
3. Business requirements
There are three key topics to discuss in the requirement collection process.
First, identify the business situation - for example maybe the churn rate started rising.
Then, assess the opportunity size, and how much you expect it to improve with machine learning. For example, you might expect to reduce churn to a certain level.
Finally, what business actions can we take? For example, do we have a strategy to save at risk customers?
4. Machine learning products
Then, we look into what machine learning products the business needs and can act upon.
Here are a few examples.
Churn prediction is a model, and the business wants two products - get a quarterly outlook of the causal churn drivers, and a daily customer classification into three groups based on their predicted churn probability - either lost, at risk or no risk customers, so the business can use these simple segmentations in structuring different campaigns.
Now an example on fraud - the fraud prediction is the model. The business might want two products - causal inference into strong indicators of churn updated monthly, and a real-time list of very risky transactions for a manual review, and a list of medium risk ones for additional data request from the customer.
5. Model performance and improvements
There's a famous saying that all models are wrong but some are useful. This is very true, all models are approximations and will make mistakes.
The business has to decide what the tolerance level is for different types of model mistakes.
In classification -
The question is which class is it more expensive to mis-classify?
For example, in the fraud prediction setting, one would imagine that it's better to mis-classify good transaction as fraud, and have them manually inspected, rather than miss fraudulent transactions and risk financial fines or worse regulatory outcomes.
In regression -
The business has to assess how much more they'll have to spend or waste resources based on the model error.
For example, in seasonal demand prediction, a packaging company will hire temporary staff, rent vehicles and buy raw materials. A high prediction error means more of these expenses, and then potentially an idle inventory, so the level of tolerated error has to be decided upfront.
6. Market testing
While we are repeating this over and over, market testing is one of the things that has to be identified early - the business has to have a list of potential market interventions and experiments it will conduct based on the model results to test machine learning readiness for production.
7. Machine learning in production
Finally, as the market tests are being implemented, we have to assess if we are getting the desired positive improvements, and if they are consistent and stable when comparing different tests over time. If yes, then it's a separate discussion on identifying production systems and resourcing the implementation of machine learning models to them.
8. Let's practice!
Great work! Let's check our knowledge on some of these communication best practices!