Why do algorithm interpretability and transparency deserve to be addressed by your data teams?

Should we always trust Machine Learning models that achieve high performance? For example, if a high-performing model recommends a specific drug to treat a patient, can we trust it blindly, or would it not be better to understand the factors that contributed to that decision?

This problem of interpretability has recently become prevalent in the AI world.

What is interpretability in machine learning?

Interpretability is the degree to which a human can understand or predict the predictions of a model. In the previous example, it would be equally important to understand the diagnosis, as well as the patient's symptoms that contributed to it. By seeing these explanations, the users' confidence in the model would increase, and it would gain social acceptance.

But the benefits of interpretability don't stop there. We use it to learn business rules from our data, to detect biases that models exploit, but also to ensure that our models do not discriminate against minorities and respect sensitive data.

Interpretability in practice at sense4data

Often, better interpretability is achieved by using simpler, less powerful models, forcing their users to choose between performance and understanding of their models. At sense4data, we prefer to use state-of-the-art techniques that allow for the interpretability of advanced machine learning models, thus allowing us to maintain the best possible performance

One of our flagship algorithms, called SHapley Additive exPlanations (SHAP), uses game theory to compute a prediction contribution score for each indicator. These scores are distributed fairly, so that a high score reflects a strong contribution and a zero score is associated with an indicator having no impact. Negative SHAP scores are also possible, reflecting indicators that go against the prediction given by the model.