MetaVoice-1B: A 1.2B Parameter Text-to-Speech Model with Advanced Voice Cloning Capabilities

MetaVoice-1B is a new text-to-speech (TTS) model from MetaVoiceIO, boasting 1.2 billion parameters and trained on an impressive 100K hours of speech data. This model stands out for its focus on emotional speech rhythm and tone in English, robust voice cloning capabilities, and support for long-form synthesis. Here’s a deep dive into what makes MetaVoice-1B a significant addition to the TTS landscape.

Key Features

• Emotional Speech Rhythm and Tone: The model is designed to capture natural speech patterns and emotional nuances without introducing hallucinations.
• Voice Cloning:
  • Finetuning with Limited Data: Successful voice cloning can be achieved with as little as 1 minute of training data, particularly for Indian speakers.
  • Zero-Shot Cloning: Supports zero-shot cloning for American and British voices using just 30 seconds of reference audio.
• Long-Form Synthesis: Capable of generating long-form content without degradation in quality.

Open Source and Licensing

MetaVoice-1B is released under the Apache 2.0 license, making it freely available for use with no restrictions. This open-source approach encourages community involvement and innovation.

Usage and Finetuning

For detailed usage instructions and finetuning guidelines, refer to the GitHub repository. The repo includes comprehensive documentation on how to get started and optimize the model for various use cases.

Architecture Details

MetaVoice-1B employs a multi-stage architecture to achieve its high-quality TTS output:

• Text-to-Token Prediction:
  • A causal GPT model predicts the first two hierarchies of EnCodec tokens from text. The text and audio are part of the LLM context, and speaker information is passed via conditioning at the token embedding layer.
    • Flattened Interleaved Prediction: Tokens from both hierarchies are predicted in an interleaved manner (first token of first hierarchy, then first token of second hierarchy, etc.).
    • Condition-Free Sampling: This technique enhances the model's cloning capabilities.
    • Custom BPE Tokenizer: A custom-trained Byte Pair Encoding (BPE) tokenizer with 512 tokens is used for text tokenization.
• Hierarchical Prediction:
  • A non-causal transformer predicts the remaining six hierarchies from the first two. This smaller model (~10M parameters) excels in zero-shot generalization and parallel prediction of timesteps.
• Waveform Generation:
  • Multi-band diffusion is used to generate waveforms from EnCodec tokens, resulting in clearer speech compared to other decoders like RVQ or VOCOS.
  • Artifact Removal: DeepFilterNet is applied post-processing to clean up any background artifacts introduced by the multi-band diffusion.

Optimizations

To ensure efficient and scalable inference, MetaVoice-1B supports:

• KV-Caching via Flash Decoding: Reduces computational overhead during inference.
• Batching: Supports batching of texts with varying lengths, enhancing performance for batched inputs.

Future Enhancements

The development team is working on several enhancements to further improve the model’s capabilities:

• Long Form / Arbitrary Length TTS: Expanding support for generating content of any length without quality degradation.
• Streaming: Enabling real-time streaming of synthesized speech.

Conclusion

MetaVoice-1B represents a significant advancement in text-to-speech technology, particularly in its ability to handle emotional speech and voice cloning with minimal data. Its open-source nature and robust architecture make it a valuable tool for researchers and practitioners alike.