Search Indexing Pipeline

Also Known As

TL;DR

Explanation

A search indexing pipeline: (1) Extract — fetch content from DB, files, or API; (2) Normalise — lowercase, strip HTML, remove special chars; (3) Tokenise — split into terms; (4) Analyse — apply stemmer (run→run, running→run), synonyms, stop word removal; (5) Index — write inverted index (term → list of document IDs with positions). Incremental indexing handles updates: updated_at timestamp queries, change data capture (CDC) from DB, or event-driven indexing via domain events. Full re-index handles schema changes.

Common Misconception

Why It Matters

Common Mistakes

• Synchronous re-indexing on every write — index updates should be async via queue.
• No incremental indexing — full re-index on every change is O(n) regardless of change size.
• Indexing HTML without stripping tags — '<strong>PHP</strong>' won't match 'PHP'.
• No synonym configuration — 'oauth' and 'open authorisation' treated as unrelated terms.

Code Examples

// Synchronous full re-index on every term save:
public function save(Term $term): void {
    $this->db->save($term);
    // Blocks request — re-indexes ALL 800 terms on every save:
    foreach ($this->db->findAll() as $t) {
        $this->searchEngine->index($t); // O(n) on every save!
    }
}

// Async incremental indexing via queue:
public function save(Term $term): void {
    $this->db->save($term);
    // Queue just this term for indexing:
    $this->queue->dispatch(new IndexTermJob($term->slug));
    // Response returns immediately — indexing happens in background
}

// Queue worker:
class IndexTermJob {
    public function handle(SearchIndex $index): void {
        $term = $this->db->find($this->slug);
        $index->upsert($term->slug, [
            'term'     => $term->term,
            'body'     => strip_tags($term->long),
            'category' => $term->category,
        ]);
    }
}

Tags

Related Terms

References

• https://en.wikipedia.org/wiki/Inverted_index