BUD-E: Advancing AI Voice Assistants with Real-Time, Empathetic Conversations

AI voice assistants have become an integral part of our daily lives, handling everything from answering queries to performing tasks. However, their responses often feel mechanical and lack the natural flow and emotional depth of human conversation. This can lead to unsatisfactory user experiences, especially when it comes to long-term interactions.

To address these issues, LAION, in collaboration with the ELLIS Institute Tübingen, Collabora, and the Tübingen AI Center, has developed BUD-E (Buddy for Understanding and Digital Empathy). BUD-E aims to enhance the conversational quality of voice assistants by enabling real-time responses, natural voices, empathy, long-term context awareness, multi-speaker handling, and local execution on consumer hardware.

Key Technical Achievements

• Low Latency: The team achieved latencies of 300 to 500 ms using a Phi 2 model, with confidence that response times below 300 ms are achievable even with larger models like LLama 2 30B.
• Real-Time Processing: BUD-E can handle real-time interactions, making conversations feel more natural and responsive.
• Natural Voices and Empathy: The assistant uses advanced Text-to-Speech (TTS) models to generate voices that sound more human-like and convey emotional intelligence.
• Context Awareness: BUD-E maintains the context of previous conversations, allowing for more coherent and relevant responses over time.
• Multi-Speaker Handling: It can manage interruptions, affirmations, and thinking pauses in multi-speaker environments.

Architecture and Implementation Details

To achieve these goals, the team focused on several key areas:

• Speech-to-Text (STT) Models: They evaluated a range of STT models to ensure low latency and high accuracy. The chosen models are optimized for consumer hardware.
• Large Language Models (LLMs): BUD-E leverages powerful LLMs like LLama 2 30B to generate context-aware and empathetic responses.
• Text-to-Speech (TTS) Systems: Advanced TTS models are used to produce natural-sounding voices. The team is working on fine-tuning these models for low-latency streaming.

Demo and Open Source

A demo of BUD-E running on a 4090 GPU is available: BUD-E Demo. All the code for BUD-E is open-source and can be found on GitHub: GitHub.

Roadmap to Empathic and Natural AI Voice Assistants

While the baseline model already provides a more natural conversational experience, there are several areas that need further development:

• Quantization: Implementing more sophisticated quantization techniques to reduce VRAM requirements and latency.
• Fine-Tuning Streaming TTS: TTS systems typically require full sentences for context. To enable high-quality low-latency streaming, the team is exploring giving the TTS system context from hidden layers of the LLM and fine-tuning it on a high-quality teacher model.

Community Involvement

LAION invites everyone to contribute to the development of BUD-E. The project is open-source, and contributions can help in refining the assistant's capabilities and making it more robust and user-friendly.

Conclusion

BUD-E represents a significant step forward in creating AI voice assistants that feel more human-like and are enjoyable to interact with over extended periods. By focusing on real-time responses, natural voices, empathy, and context awareness, BUD-E aims to bridge the gap between current voice assistants and truly immersive conversational experiences.