A Chatbot Coaching Case Study Revealing Surprising Results
A case study video from my mum’s English garden. Enjoy!
A first for me – A case study vlog! This is the case study that I shared at the NYU (New York University) Coaching and Technology Summit last month. The study, conducted over a nine-month period, explores the effectiveness of chatbot Coach M to support performance goals.
Client-Driven Innovation:
At Lever, we thrive on client-driven innovation. While we pride ourselves on being innovative, our clients consistently challenge us to go beyond and explore new possibilities. In this particular project, spanning nine months, we collaborated closely with a client to support the embedding of action plans within their regular performance process using chatbot coaching. It was a first to be using Coach M beyond the normal setting of the embedding of action plans from a training program to supporting annual performance goals.
Control Groups and Data Comparison:
To ensure a comprehensive analysis, we were fortunate to have the client’s permission to conduct a study with two control groups. The first group (Coach M and Human group) received three (3) 30 minutes of human coaching along with chatbot coaching support. The second group (Coach M only group) solely relied on Coach M’s assistance throughout the nine-month period. By comparing the data between these two groups, we were able to review the impact of human coaching versus chatbot coaching.
Goal Progression:
One of the key metrics we examined was goal progression, where participants self-rated their progress at the beginning and end of the assignment. Interestingly, Coach M held its ground impressively, with just a 2-percentage point difference between the Coach M and Human group. While it’s important to acknowledge that variations in initial goals may have influenced the scores, it is evident that utilising Coach M in supporting goal progress is a valuable standalone measure. The data revealed an 84% uplift in goal progress for the Coach M and Human group, while the Coach M-only group experienced an 86% uplift.
Perceived Progress:
To gauge participants’ perceived progress, we asked them to assess the level of progress they felt they had made on their goals during the study period. Comparing the two intervention groups, we discovered that using Coach M alone resulted in a remarkable 35% uplift in perceived progress. This figure closely aligns with findings from other groups employing Coach M for learning transfer. On the other hand, the human coaching group reported a significant 92% differential in progress on their goals. What does this tell us? It shows that emphasising the value of human coaching for significant and transformative change, but that for making progress on goals, Coach M has a role to play.
Augmented Process and Seamless Coaching:
Throughout the project, we integrated Coach M seamlessly into the coaching process, allowing for a three-way coaching experience. The human coach had visibility into the conversations between the individual and Coach M, while Coach M gained insights from the human coach’s interactions. As the coach on this project, I found that having Coach M handle some of the surface-level progress questions left me as the coach to be able to go deep in a short period of time with the human coaching.
Recommendation Scores:
We also measured participants’ recommendation scores for working with the coaching tool. The Coach M-only group had an average recommendation score of 7.8, highlighting their satisfaction with the tool’s ability to assist in following through on action plans. In comparison, the human coaching group received an average recommendation score of 9.7 for the human coaching component (pat on my own back!) and 6.5 for the Coach M component. Considering the direct comparison with human coaching, is a commendable result.
Conclusion and Future Research:
I’m delighted with the outcomes of this case study, which demonstrate the power of chatbot coaching in goal progression. While human coaching excels in driving transformative change, Coach M proves highly effective in keeping goals on track and it is scalable! The study highlights the immense value of coaching and our dedication to remaining at the forefront of this work.
As we wrap up this project, we are excited to announce that more research is on the horizon. Stay tuned for an upcoming announcement. Gratitude as always to our clients for their continuous support and collaboration, which fuels our ongoing innovation.
If you are considering projects for the immediate future or planning ahead for 2024, I would love to hear from you. Our team is ready to help you explore interventions, create impactful data for your organisation, and make a difference in performance as a learning professional. Please feel free to book a time in my diary. Just note that I’ll be away for a couple of weeks and will be back in August to take some time off.
Wishing you all the very best and looking forward to those upcoming conversations.