Study Reveals Linear Relationship Between Compression Efficiency and LLM Intelligence

In a recent study titled "Compression Represents Intelligence Linearly," researchers from Hong Kong University of Science and Technology (HKUST) have provided empirical evidence that the ability of large language models (LLMs) to compress external text corpora correlates almost linearly with their intelligence. This work, published in COLM 2024, sheds light on why advanced compression capabilities are a strong indicator of a model's performance across various benchmarks.

What Changed Technically and Why It Matters

The study challenges the common belief that intelligence is an abstract, hard-to-quantify concept by showing a clear, measurable relationship between compression efficiency and LLM performance. This has significant implications for practitioners:

• Compression as a Proxy for Intelligence: The researchers used average downstream benchmark scores to represent "intelligence" in LLMs, focusing on knowledge, commonsense, coding, and mathematical reasoning. They found that the better an LLM can compress external text corpora, the higher its performance on these benchmarks.
• Unsupervised Evaluation Metric: Compression efficiency, derived from raw text data without labeled supervision, serves as a reliable and linearly associated evaluation measure for model capabilities. This opens up new avenues for unsupervised model assessment.

Key Findings

The study involved 31 public LLMs from diverse organizations across 12 benchmarks. Here are the main findings:

• Linear Correlation: The average benchmark scores of LLMs were found to almost linearly correlate with their ability to compress external text corpora. This suggests that improving compression capabilities directly enhances model intelligence.
• Diverse Benchmarks: The study targeted a wide range of tasks, including:
  • Knowledge and Commonsense (e.g., TriviaQA, ARC)
  • Coding (e.g., HumanEval, MBPP)
  • Mathematical Reasoning (e.g., MATH, GSM8K)

Methodology

To establish this relationship, the researchers:

• Collected Datasets: They compiled a diverse set of external text corpora and benchmark datasets.
• Compression Evaluation: Used standard compression algorithms to measure how well each LLM could compress these corpora.
• Benchmark Scoring: Calculated average scores across 12 benchmarks for each LLM.

Implementation Details

The study's methodology is transparent and reproducible:

• Data Collection Pipelines: The researchers open-sourced their data collection pipelines, making it easier for future researchers to replicate the study.
• Code Availability: All datasets and code are available at this GitHub repository.

Implications for Practitioners

For developers and researchers working with LLMs, this study offers several practical insights:

• Model Evaluation: Compression efficiency can serve as a reliable unsupervised metric to evaluate model capabilities.
• Development Focus: Enhancing compression algorithms could lead to more intelligent models, potentially improving performance across various tasks.
• Benchmarking: The study's comprehensive approach to benchmarking provides a robust framework for assessing LLMs.

Conclusion

The findings from "Compression Represents Intelligence Linearly" provide concrete evidence that the ability of LLMs to compress external text corpora is a strong indicator of their intelligence. This opens up new possibilities for unsupervised model evaluation and suggests that focusing on compression could be a key strategy for developing more advanced language models.