Does AI coaching move performance?

‍This originally ran in AI & the Workforce, Valence's newsletter on how AI is changing leading enterprises and the way their people work, lead, and grow. Subscribe here.

Most organizations can describe adoption in detail. Licenses activated, logins counted. Adoption matters, but it only tells you people showed up. And for all the data enterprises now have about AI, one question has stayed stubbornly out of reach: what is it doing to how people actually perform?

That is the question the two of us set out to answer. In the first study from the Valence Research Initiative (VRI), we followed 11,000 employees at a large technology company through a full performance review cycle. Because Nadia, Valence’s AI coach, was embedded in the review process itself, we could do something that has rarely been possible in enterprise AI: compare people's actual performance ratings before an AI tool arrived and after, against how deeply each person engaged with it.

We found power users had 31% higher odds of moving up a performance band. Across every level and part of the organization, the more intensely and deeply someone engaged as a power user, the bigger the gains.

We created the AI Power User Index to benchmark AI engagement. Frequency matters, but so do breadth and depth of use. We designed the AI Power User Index to quantify not just how often someone uses the tool, but also the variety of situations people brought to the coach and how much of their own thinking they put into the exchange.

It’s a methodology that can be applied to any AI initiative, and we’ll unpack in a future edition how any leader can apply it beyond coaching.

When AI is embedded into talent processes, we create the opportunity for broad and deep engagement. By then connecting engagement, through the AI Power User Index, to annual performance results, we can directly see if and how AI drives real performance gains.

Engagement predicts impact, and managers are the lynchpin to drive engagement. Managers were 3x more likely to be power users, and when managers were power users, their teams were 2x more likely to be.

You can find the full research findings here: valence.co/performance-research

‍

About the Authors

Prasad Setty chairs the Valence Research Initiative and previously led People Analytics at Google, where his team ran Project Oxygen and Project Aristotle. 

Jennifer Carpenter is a global HR leader who has led large-scale talent transformations at Accenture, Delta Air Lines, IBM, and Analog Devices.

‍