Zoom's AI Evolution: From Meetings to Milestones with Federated LLMs and Custom Agents

Zoom, the video conferencing giant that became a household name during the pandemic, is no longer content to just facilitate meetings. The company has been quietly building an advanced AI infrastructure over the past few years, aiming to transform its platform into a comprehensive productivity tool. This shift from meeting facilitation to action-oriented workflows is a significant step forward in the agentic AI era.

Moving from Meeting to Milestone

Meetings are a staple of business communication, but their true value lies in what they achieve-action items, decisions, and milestones. Zoom CTO Xuedong (X.D.) Huang sees this transition as a pivotal moment for technology:

"In the agentic AI era, finally technology is reaching the point that we can transform from meeting to milestone," Huang told VentureBeat in an exclusive interview.

Zoom's latest update introduces a suite of new services designed to make meetings more productive. These include agentic capabilities that convert meeting content into actionable workflows and an AI Studio for enterprises to create customized AI agents.

The Technical Evolution Behind Zoom's Agentic AI

Huang, who spent 30 years at Microsoft working on speech technologies and Azure OpenAI, brought a wealth of experience to Zoom when he joined in 2023. Under his leadership, the company has developed an AI architecture focused on facilitating tasks rather than just summarizing conversations.

• Federated Approach to LLMs: Unlike competitors like Microsoft Teams (which relies on OpenAI via Azure) and Google Meet (supported by Google Gemini), Zoom has taken a federated approach to large language models (LLMs). This means they use multiple LLMs, including:
  • Partnerships with leading models such as OpenAI and Anthropic
  • A custom-built small language model (SLM) with 2 billion parameters

• Custom AI Companion: When Zoom launched its first AI companion in 2023, it wasn't tied to a single LLM. Instead, the company leveraged a combination of external models and their own SLM to create a versatile AI assistant.

"We've partnered with the best models out there, including OpenAI and Anthropic, but we've also built our own highly customized 2 billion parameter language model," said Huang.

Implementation Details

• Agentic Capabilities: These features are designed to automatically identify action items, summarize key points, and suggest next steps during meetings. They can also integrate with project management tools to ensure that meeting outcomes are actionable.
• AI Studio: This new tool allows enterprises to create custom AI agents tailored to their specific needs. For example, a legal firm might create an agent specialized in contract review, while a marketing agency could develop one for content generation.

Benchmarks and Performance

While Zoom hasn't released detailed benchmarks yet, early feedback from beta users has been positive. The agentic capabilities have shown promise in reducing meeting follow-up time by automating task assignments and summarizing key discussions. The AI Studio is also gaining traction among enterprise customers who appreciate the flexibility to tailor AI agents to their workflows.

Conclusion

Zoom's transition from a video conferencing tool to an agentic productivity platform marks a significant evolution in how technology can enhance workplace collaboration. By leveraging a federated approach to LLMs and custom-built models, Zoom is positioning itself at the forefront of this new era of AI-driven productivity.