… along with use-case examples for easier stakeholder collaboration
Table of Contents
Introduction
While there are a ton of data science or specific algorithm metrics like MAE and RMSE that are useful to know, there are other metrics that can mean more to stakeholders and your business as a whole.
It is often not taught as much in academia, but it is just as important to know, practice, and employ. For those reasons, I am going to look into three use cases where you can benefit from utilizing business metrics for collaboration with stakeholders (especially the ones who are not in data science).
Bucketed Unit Ranges
It is important to keep in mind that not all of these metrics will be applicable to all use cases, so that is why I am giving an example of one so that you can determine if it is suitable for your situation.
With that being said, let’s first discuss a use case where you are predicting a continuous target, like numbers from 1–100.
In the current space, you can use MAE or RMSE, for example, to understand how you’re specific model is doing. However, when talking with executives or other stakeholders, you will want to explain the error in terms of how it affects the business.
The use case example would be to predict the amount of money a house would be in order to put it up for sale. An expected target would be anywhere, say, in the range of $200,000–$300,000. After establishing your features and building your model, you will see that your MAE is $10,000 and your RMSE is $30,000. While this might sound great, and be useful to your model, it might not help your stakeholders make further business decisions.
Some of business metrics that you could instead are the following — essentially from bucketed dollar amount ranges:
- % of predictions that were $10,000 less than the actual →
ex: 40% of predictions were within $10,000 less than the actual, another way to look at this business metric is how many of our predictions were underestimating when compared to the actual
- % of predictions that were $10,000 above the actual →
ex: 20% of predictions were within $10,000 greater than the actual, another way to look at this business metric is how many of our predictions were overestimating when compared to the actual
The reason you would want to use this business metric approach is that stakeholders, and yourself, could be able to know how much you are off by on either side of the prediction. The limitation of MAE for example, is that it is looking at predictions below and above the actual (as indicated by the mean absolute error) at an aggregate level. Depending on your use case, you may want predictions to inherently overestimate or underestimate.
Change in Impact
Now that we have a sense of how error can be interpreted differently, we can look at the impact of what the previous metric can mean for the business. For example, when looking at the number of predictions in a bucketed range of money, we will want to consider what it means for the business.
Here are some examples and discussions around change in impact business metrics:
- Overestimating leads to fewer sales
ex: what % of sales took longer than two weeks when we do or do not overestimate?
- Underestimating leads to less money for sellers
ex: what % of offers were made at the underestimated amount?
As you can see, this use case is specific to housing sales, but it can be applied to different scenarios. For example, car sales, any product sales, and so on. It does not even need to be pertaining to sales, it could be something like how many logins occurred for your company app when you were shown a certain group of predictions.
The impact of your predictions is usually centered around an A/B test, or a comparison of what happens when you present certain predictions, and what about your business is most important to you or your stakeholders, along with if those metrics have changed significantly.
Summary
With these two main examples of types of business metrics, we can see how just discussing the MAE, MAPE, AUC, etc. is not sufficient enough when incorporating a model into an official release for a business. It is always best to test the model in terms of the business, and we have discussed that bucketed metrics, as well as their impact on product and user behavior, can work to understand the business better.
To summarize, here two important business metrics you should know:
* Bucketed Unit Ranges* Change in Impact
I hope you found my article both interesting and useful. Please feel free to comment down below if you agree or disagree with these examples of important business metrics. Why or why not? What other metrics do you think you could use or are important? These can certainly be clarified even further, but I hope I was able to shed some light on some more unique and specific business metrics for data scientists and stakeholders. Thank you for reading!
