Claude's Tokenizer Breakthrough: Smaller Vocab, Better Performance?

Tokenizer enthusiast Sander Land recently reproduced something very similar to Claude’s current tokenizer, which appears to have only about 15,000 entries. This is quite surprising, especially when compared to Qwen 3.8, a highly regarded model with around 250,000 tokens in its vocabulary. The trend in the industry has generally been that more tokens lead to better performance, so this deviation from the norm is intriguing.

One theory behind this unconventional approach is that Anthropic may be addressing a bottleneck caused by the final softmax layer during backpropagation. A recent paper, "Lost in Backpropagation: The LM Head is a Gradient Bottleneck," delves into this issue. According to the authors, the problem arises because you need to project from the model’s latent space (dimension D) to a much larger vocabulary space (dimension V) to select a token.

Understanding the Gradient Bottleneck

During training, the model compares its predicted distribution of tokens to the actual next token. If the model is correct and confident, the loss is small; if it's confidently wrong, the loss is large. This loss then propagates through all 250,000 entries in the vocabulary space and down to the 8,192 entries in the hidden dimension (D) for Qwen’s 2.4T parameter flagship model.

The compression of this information during backpropagation can create a bottleneck. Specifically, the change in logits has a rank at most 2D. If the vocabulary size V is significantly larger than D, a lot of information is lost:

• Empirical and Theoretical Evidence: The authors demonstrate both empirically and theoretically that the softmax bottleneck causes lossy compression during backpropagation.
• Sparse Signals: While the correct distribution is just one entry wide (the actual next token), over a batch, you get various different next tokens. These signals are sparse but not low rank, meaning nearly every token can be "the next token" at some point.

This means the learning signal coming in is as wide as the vocabulary, and the model samples a random D-sized subset of it. While this isn’t inherently problematic, there’s no particular mechanism that encourages the model to learn the optimal mapping between the hidden dimension and the vocabulary space.

Key Takeaways

• Smaller Vocabulary: Anthropic's Claude has reduced its vocabulary size from around 50,000 entries in Claude 3 to about 16,000 today. This suggests they are addressing the gradient bottleneck issue.
• Efficiency and Performance: By reducing the vocabulary size, Anthropic might be avoiding the gradient bottleneck, leading to more efficient training and potentially better performance.
• Industry Trends: Most models have been increasing their vocabulary sizes, but Claude’s approach shows that smaller can sometimes be better, especially when dealing with large-scale models.

Whether this is the primary reason for the small vocab or not, it’s clear that Anthropic has found a way to mitigate the costs associated with the gradient bottleneck. This approach also offers several other benefits, such as reduced computational overhead and potentially faster training times.

In practice, this could mean that future models might start exploring smaller vocabulary sizes to improve efficiency and performance. As researchers continue to push the boundaries of what’s possible in natural language processing, it will be interesting to see how these insights are applied and whether they lead to more widespread adoption of similar techniques.