What’s The Score? Developing The Right Measurement Capability Is Critical To AI Success
Steve Gustafson
Steve Gustafson

Steven Gustafson is Chief Scientist overseeing research and data science at Maana.

22

March 2018

What’s The Score? Developing The Right Measurement Capability Is Critical To AI Success

Several years ago, I was involved in defining an artificial intelligence (AI) system to improve help desk tickets for a large IT provider. They received hundreds of tickets per hour across a global customer base. The leadership identified a key question for the AI system to answer: Given a new IT problem by a user, what is the first resolution they should attempt?

Initially, customers wanted us to generate a recommended set of actions to resolve new problems by mining previous cases and solutions. After several interviews and discussions, we identified a new set of related challenges. Despite having a global network of employees solving similar tickets at the same time, employees were having difficulty properly labeling new tickets and their successful solutions consistently. This labeling challenge made those tickets either difficult to find or invisible to other help desk employees at the time when they would be most helpful — solving a nearly identical ticket occurring in the same time period.

We also discovered it was difficult to measure outcomes of recommendations the employees made. Sometimes tickets were not closed correctly or lacked important metadata to understand if all the recommended actions were necessary and correct. This experience taught us a vital lesson about what dictates the success of AI projects.

Develop A System Of Measurement
As a leader in AI research & development (before joining Maana as the chief scientist, I led the Knowledge Discovery Lab at the General Electric Global Research Center, focusing on knowledge graphs and machine learning), I have been asked several times to explain AI and the ways it provides value to large industrial companies. Despite many differences between companies, the common goal of this type of AI remains the same: to improve productivity within an organization by allowing employees to make better, more informed decisions.

Conversations about AI often focus on the potential and certainty of an outcome the AI solution can deliver (e.g., decrease customer IT tickets by 10% or decrease the time it takes to close IT tickets by 20%?). However, these overall business goals may not directly align to the type of measurement the AI system will need to be successful: What specific actions did a customer follow to resolve an IT issue, and which of those were successful?

We wanted an AI system to improve the IT ticket resolution process and provide better customer and employee satisfaction. The initial system design was toward recommending steps to resolve the issue, but the success of that system would be predicated in part upon measuring the outcomes of the employee recommendations. However, we struggled to find an immediate path to measuring the outcome of those recommendations directly, rendering the design of the AI system incapable of improving the ticketing process.

Being successful in AI applications requires solving this joint problem of finding the most effective outcome — for the business and customers — that also has the data and measurement capability needed by an AI approach. Without the ability to measure the recommendation outcomes, an AI system will fail in the long term or become very costly to maintain.

Combine Subject Matter Expertise With Data

To create a better AI solution, you need to leverage the domain expertise of IT employees with the data that a system collects. In our situation, we realized:

  • Employees were primarily able to store and retrieve their domain knowledge using labels given to previously solved IT issues.
  • They struggled to assign labels during the resolution process that were consistent across all their employees.

The solution was an AI system that would recommend a label, intermittently gather feedback from the employee about whether the label was correct and use the feedback to continually improve the labeling AI system. This solution was met with positive feedback from the IT employees, as it allowed them to still apply their experience and technical knowledge while assisting them at the tedious tasks of selecting the best label for every new ticket. It also had a future advantage of preventing new tickets by allowing IT engineers to see trending issues better and head them off, as well as enabling customers to search for solutions themselves.

Tell The AI System The Score
Imagine that instead of optimizing IT tickets, you’re developing an application for field engineers at a large energy company. Their goal is to keep critical infrastructure working, in part by prioritizing which pieces of equipment at which customer locations require inspection or repair based on maintenance schedules or predictive maintenance algorithms. In this opportunity, the number of possible objectives and measurements increases significantly to encompass the health of equipment, the efficiency of servicing support contracts, satisfaction and profitability of the customer and overall productivity and satisfaction of the field engineer.

For this scenario, like many other industrial scenarios, success is often not as easily measurable for an AI system. Overall, a company tries to make an objective — but still often subjective — decision as to the success of its maintenance actions and productivity of its employees. Businesses develop competitive business models and processes that benefit directly from the domain knowledge and experience of employees. The experience-based knowledge allows businesses to differentiate and provide ever-improving capability to customers. Thus, when developing an intelligent solution using AI, organizations must begin by considering the following:

  • Understand what business question the AI system is answering and how you will confidently measure the outcome.
  • Identify how the AI system can complement the expertise of its users, allowing it to gather feedback and improve over time.

Pinning those subjective and experience-based decisions down into concrete data and measurements that an AI algorithm could make use of is often very difficult. Therefore, being able to tell an AI system the “score” based on a measurement capability — so it knows if it is winning or losing and can learn — is the first step toward achieving value from AI. Combining that score with feedback from users based on their experience and domain knowledge allows the AI system to improve over time.


by Steve Gustafson in Maana Leadership