sqlite-vec v0.1.0: A Portable Vector Search SQLite Extension for Multiple Languages

The first stable release of sqlite-vec v0.1.0 is now available, bringing vector search capabilities to SQLite in a lightweight and highly portable manner. Written entirely in C with no external dependencies, this extension is MIT/Apache-2.0 dual licensed and can be installed across multiple package managers for various programming languages.

What Changed Technically

sqlite-vec introduces a new way to perform vector searches within SQLite databases. Here’s what makes it significant:

• Single C File: The entire extension is contained in one C file, making it easy to compile and integrate.
• No Dependencies: It has no external dependencies, ensuring that it can run almost anywhere without additional setup.
• Multi-Language Support: It supports Python, Node.js, Ruby, Rust, and Go (both CGO and non-CGO WASM flavors).

Why It Matters

For developers working with SQLite, sqlite-vec offers a powerful tool for handling vector data. This is particularly useful in applications involving machine learning, natural language processing, or any scenario where you need to perform similarity searches on high-dimensional vectors.

Installation Options

You can install sqlite-vec using several package managers:

• Python:

  pip install sqlite-vec
  

• Node.js (Node, Bun, Deno):

  npm install sqlite-vec
  

• Ruby:

  gem install sqlite-vec
  

• Rust:

  cargo add sqlite-vec
  

• Go (CGO):

  go get github.com/asg017/sqlite-vec-go-bindings/cgo
  

• Go (Non-CGO WASM):

  go get github.com/asg017/sqlite-vec-go-bindings/ncruces
  

• Direct Installation (for the adventurous):

  curl -L https://github.com/asg017/sqlite-vec/releases/download/v0.1.0/install.sh | sh
  

How It Works

sqlite-vec operates similarly to SQLite's full-text search support, using virtual tables for vector columns. Here’s a quick example:

-- Create a virtual table with vector columns
create virtual table vec_articles using vec0(
  article_id integer primary key,
  headline_embedding float[384]
);

-- Insert data into the virtual table
insert into vec_articles(article_id, headline_embedding) values
   (1, '[0.1, 0.2, ...]'),
   (2, '[0.3, 0.4, ...]'),
   (3, '[0.5, 0.6, ...]');

-- Perform a KNN-style query to find the 20 closest headlines to 'climate change'
select
	rowid,
	distance
from vec_articles
where headline_embedding match embed('climate change')
  and k = 20;

Architecture Details

• Virtual Tables: vec0 virtual tables store vectors in shadow tables, similar to how fts5 handles full-text search. This ensures efficient operations during inserts, updates, and deletes.
• KNN Search Optimization: The MATCH constraint on vector columns triggers a K-nearest neighbors (KNN) search, which is optimized for speed.

Future Plans

sqlite-vec is expected to be integrated into popular SQLite-related products like SQLite Cloud and Turso. This will make it even more accessible and useful for developers working with cloud-based or managed SQLite services.

Try It Out Today!

With its lightweight design and broad language support, sqlite-vec is a versatile tool for adding vector search capabilities to your projects. Give it a try and see how it can enhance your data handling and query performance.