Building a Web Search Engine with Transformers and Neural Embeddings

A few months ago, Wilson Lin embarked on an ambitious project: building a web search engine from scratch. His primary motivations were the declining quality of search results due to SEO spam and the impressive advancements in transformer-based text embedding models. Lin’s goal was simple yet challenging: create a search engine that consistently returns top-quality content, leveraging modern NLP techniques to understand complex and subtle queries.

Technical Overview

Key Motivations

• Declining Search Quality: Modern search engines often return low-quality, SEO-spammed results.
• Advancements in NLP: Transformer models like SBERT have shown remarkable language comprehension capabilities.

Lin’s project aimed to combine these insights to build a more intelligent and effective search engine. Here are some of the key technical highlights:

Data Ingestion

• Crawling: A fleet of hundreds of crawlers ingested 50,000 pages per second, resulting in an index of 280 million documents.
• Queue System: A custom RocksDB-based queue handled 300,000 operations per second, ensuring efficient data processing.

Text Processing

• Parser: The system used the WHATWG spec for semantic extraction to parse and understand the content.
• Normalizer: Chrome-specific elements were removed, and content was canonicalized for consistency.
• Chunker: spaCy's sentencizer was employed to break down text into manageable chunks while preserving context.
• Statement Chainer: A DistilBERT classifier resolved anaphora, enhancing the coherence of processed statements.

Compute Resources

• GPU Cluster: 200 RTX 4090 GPUs generated over 3 billion SBERT embeddings at a rate of 100,000 embeddings per second with 90% utilization.
• CPU Processing: Real-time query vectorization was handled by the CPU to generate embeddings for incoming queries.

Data Storage and Indexing

• Distributed RocksDB: The system used 64 shards across 82 TB of SSDs, capable of handling 200,000 writes per second.
• Vector Index: A distributed HNSW (Hierarchical Navigable Small World) index with 64 shards and 4 TB of RAM stored 3 billion embeddings.

Query Latency

• The end-to-end query latency was optimized to around 500 ms, ensuring fast and responsive search results.

Architecture

The high-level architecture of the system can be broken down into several key components:

1. Edge
   • Frontend: Utilizes Cloudflare for Argo routing and HTTP/3 support.
2. App
   • Streaming SSR: Ensures a 500 ms latency for query processing.
   • SERP (Search Engine Results Page): Displays search results, knowledge panels, and an AI assistant.
3. Ingestion
   • Crawling Fleet: Ingests 50K pages per second using multi-level stochastic queues.
4. Queue System
   • Custom RocksDB Queue: Handles 300K operations per second for efficient data management.
5. Text Processing
   • Parser, Normalizer, Chunker, Statement Chainer: Processes and prepares text for indexing.
6. Compute Resources
   • GPU Cluster: Generates embeddings using SBERT models.
   • CPU: Handles real-time query vectorization.
7. Data Storage and Indexing
   • Distributed RocksDB: Stores records and blobs with high write throughput.
   • Vector Index: Uses HNSW for efficient similarity search.

Live Demo

You can try out a live demo of this search engine to see how it performs in real-world scenarios. The project showcases the potential of combining advanced NLP techniques with robust infrastructure to create a more intelligent and user-friendly search experience.