Thinking Machines Releases Inkling-Small: A Compact, High-Performing AI Model

Thinking Machines, the well-funded startup led by former OpenAI CTO Mira Murati, has once again made waves in the AI community with the release of Inkling-Small. This new model is a significant step forward, offering developers a compact yet powerful alternative to the original Inkling. Despite being about one-fourth the size, Inkling-Small maintains impressive performance across various benchmarks.

Key Features and Performance

Inkling-Small is a 276-billion-parameter multimodal reasoning model released under the permissive Apache 2.0 license. It supports text, image, and audio inputs, producing text outputs with a context window of up to one million tokens. Here are some key details:

• Parameter Count: 276 billion parameters (compared to 975 billion in the original Inkling)
• Active Parameters per Token: 12 billion (compared to 41 billion in the original)
• Benchmarks:
  • On the third-party Artificial Analysis Intelligence Index, Inkling-Small scores a 40, just one point below the original Inkling's score of 41.
  • The model preserves much of the flagship’s coding, reasoning, and multimodal performance.

## Under the Hood

The technical architecture of Inkling-Small is designed to balance efficiency with high performance. Here are some implementation details:

• Model Architecture: Inkling-Small uses a transformer-based architecture optimized for efficient inference and training. The model leverages techniques like sparse attention and mixed-precision training to reduce computational overhead.
• Parameter Efficiency: By using 12 billion active parameters per token, the model significantly reduces the compute requirements compared to its larger counterpart, which uses 41 billion active parameters.
• Context Window: Supporting a context window of up to one million tokens allows for more extensive and nuanced inputs, making it suitable for complex tasks like long-form content generation and deep reasoning.

## In Practice

For enterprises and developers, the appeal of Inkling-Small lies in its ability to deliver high performance with reduced resource requirements. Here are some practical considerations:

• Compute Requirements: While still too large for a laptop or conventional workstation, Inkling-Small is more feasible for deployment on modest GPU clusters.
• Inference Costs: The model's smaller size translates to lower inference costs and faster processing times, making it an attractive option for resource-constrained environments.
• Deployment Footprint: With a 3.5X reduction in size compared to the original Inkling, Inkling-Small is easier to deploy and manage, especially for organizations with limited GPU resources.

Thinking Machines has made the full weights of Inkling-Small available on Hugging Face, along with support for fine-tuning through its Tinker model training API. To encourage adoption, the company is offering a limited-time 50% discount on API pricing:

• Standard 64K-context Model:
  • $0.58 per million prefill (input) tokens
  • $1.44 per million sampled (output) tokens
  • $1.73 per million training tokens
  • Cached prefill requests: $0.116 per million tokens
• 256K-context Variant: Available at higher rates

Artificial Analysis, a third-party benchmarking service, has assigned Inkling-Small a score of 40 on its Intelligence Index, just one point below the original Inkling's score of 41. This result underscores the model's effectiveness and potential for widespread adoption in various applications.

Thinking Machines' release of Inkling-Small marks a significant advancement in the field of compact, high-performing AI models. With its balanced approach to performance and efficiency, Inkling-Small is poised to become a valuable tool for developers and enterprises looking to leverage advanced AI capabilities without the burden of excessive resource requirements.