Measuring the POC
1. Measuring the POC
Welcome back! We've come a long way, learning about AI, how to use it, how to build a POC, and more. Now, we will learn about how to define and measure success.2. How to define success?
Ultimately, specific criteria will depend on the use case and goals. They will likely fall within these categories - business value,3. How to define success?
AI model performance,4. How to define success?
explainability,5. How to define success?
and solution performance. A successful POC is one that achieves the desired threshold for metrics from these categories.6. Measuring business value
Business value is often one of the primary ways to measure success. This will be defined by the POC and business objectives. Again, this will typically fall into the buckets of efficiency, productivity, creativity, and skilling.7. Measuring business value
As an example, let’s take an AI virtual agent supporting scheduling clients. The overall objectives may be “reduce customer service agent hours spent on scheduling” which would allow employees to focus on higher-level tasks, as well as increase the number of scheduled clients. Some metrics to monitor are agent time spent scheduling, number of appointments scheduled, percentage of interested clients scheduled, time to booked appointment, percentage of canceled appointments rescheduled (for those that canceled because of conflicts). The other crucial factor to success will be the performance of the AI model.8. Measuring the AI model
Two ways to measure the performance of the AI model, among many, are accuracy and relevance.9. Measuring the AI model
Accuracy is the percentage of predictions an AI solution correctly suggests. Going back to the example, a high accuracy indicates the AI solution is suggesting suitable time slots based on past appointments with the customer or customers like them. Accuracy is a go-to metric for measuring AI performance. There are many other metrics, however, which may be more suitable for the particular business problem a solution is solving for. Consult an ML scientist to determine the right one.10. Measuring the AI model
Relevance is closely related and focuses on providing results that are expected based on the input - both direct and indirect. For example, the AI scheduling agent should provide relevant suggestions based on the customer’s timezone as well as offices closest to their location.11. Explainability of the AI
Explainability is the ability to understand the AI model’s decision-making process. Measurements here will focus on fairness, transparency, bias, and safety. In our example, fairness could be measured by making sure the AI provides appropriate and optimal times for all customers instead of those who spend large amounts with the company. Transparency helps build trust with AI solutions. User-feedback can help measure this, asking users if they understand why a time was suggested. Transparency also requires intentional design as explainability frameworks will need to be built into the AI model architecture.12. Measuring solution performance
Metrics focused on solution performance are used to evaluate the components outside the AI model but may still influence the model performance and its usability. They have a strong negative effect on business value if not performing well.13. Measuring solution performance
An unusable solution14. Measuring solution performance
is a costly and valueless solution.15. Measuring solution performance
Going back to our example, solution performance can be evaluated by understanding the following. First, latency and real-time updates. The AI model should only make suggestions for open time slots. Therefore, the solution needs to be fast enough to send booked times to the AI model. Not doing so means an increase in rescheduling and negative customer reviews. Similarly, the UI is important. It needs to be responsive to customer use and input. Some metrics would be load time, time to appointment confirmation, and customer satisfaction (through surveys).16. Let's practice!
These are just some ideas to get started with defining success and what to do after the POC!Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.