The concept of voice as a primary interface is gaining significant traction, attracting billions in investments for AI voice startups spanning various applications, including model development, enterprise customer service, meeting transcription, and AI dictation.
Despite frequent new model and tool releases purporting human-like speech and conversation, true natural interaction remains elusive. Shawn Wen, CTO of enterprise voice AI platform PolyAI, believes that even with the advent of full-duplex models
— which allow simultaneous speaking and listening
— voice AI has yet to experience its breakthrough “ChatGPT moment.”
“While we’ve achieved full-duplex models, the subsequent hurdle is to accelerate reasoning capabilities, enabling models to retrieve answers rapidly and foster more natural conversations,” he explained at the HumanX conference last month.
Wen emphasized that customer service AI agents must avoid robotic tones and instill confidence in callers regarding their problem-solving abilities.
“The next phase will diverge slightly. Once the voice quality is sufficient, and customers are willing to interact for a few exchanges, they’ll gain confidence, eventually feeling that a human isn’t necessary if the AI agent can resolve their issue,” he stated.
Alex Gay, CMO of Otter, a meeting notetaker, asserted that accurate speaker identification, intent recognition, and integrating these with organizational knowledge are crucial for effective automation. Otter is also developing “digital twins” to represent individuals in meetings, stressing the importance of these avatars conveying the same emotional expressions as human participants.
“Consider your current meetings: the most productive conversations involve debate, strategic discussions, and a foundational relationship. Without the ability to foster that with an avatar, it merely becomes a Q&A chatbot,” Gay observed.
Understanding and Transparency in Voice AI
Despite advancements in voice AI models, AI assistants frequently struggle with user comprehension, leading to inaccurate transcripts or summaries from meeting notetakers.
Wen believes ASR (Automatic Speech Recognition) models frequently overlook crucial keywords, hindering their ability to capture complete conversational context.
Otter’s Gay concurred, noting the company’s continuous efforts to enhance transcription accuracy and highlighting language as another domain requiring voice model improvement.
“For Otter, transcription was never the ultimate goal; rather, it served as the foundational layer for achieving productivity gains. However, inadequate accuracy in the initial transcription compromises all subsequent actions. When incorrect actions are taken, trust in the platform is lost. Therefore, continuously enhancing the ASR model is crucial due to its significant downstream effects,” he explained.
The emergence of new voice tools also raises concerns about transparency. These tools ought to clearly inform users if they are being recorded or interacting with an AI. Otter aims to build trust among meeting participants, proposing notification methods
— even in the absence of a bot
— to inform everyone via chat that a meeting is being recorded. PolyAI’s Wen similarly stressed the importance of clearly identifying AI interactions in enterprise calls.