← Back to DevBytes
Web July 12, 2026 19 min read

Implementing Web Content Indexing in Modern Web Applications

Understanding Web Content Indexing

Web content indexing is the process of systematically cataloging the textual content, metadata, and structured data within a web application to enable fast, accurate search and retrieval. Unlike traditional database queries that scan records sequentially, an index pre-organizes content into lookup structures—much like the index at the back of a reference book—allowing near-instantaneous results even across millions of documents.

In modern web applications, indexing operates at multiple levels: client-side indexing for offline-first and single-page applications, server-side indexing backed by dedicated search engines, and hybrid approaches that synchronize both. The core principle remains the same: extract meaningful tokens from your content, map those tokens to their source locations, and store that mapping in a structure optimized for retrieval.

Key Terminology

Why Content Indexing Matters

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

Without indexing, search functionality degrades to a linear scan—a brute-force approach that becomes unusably slow as content grows. Consider a documentation site with 50,000 articles. A naive string-matching search might take several seconds per query, consuming significant CPU resources and delivering a poor user experience. An indexed search completes the same operation in milliseconds.

Beyond raw speed, indexing enables capabilities that simple pattern matching cannot provide:

For offline-capable Progressive Web Apps, client-side indexing is essential—it allows search to function even when the network is unavailable. For content-heavy platforms like e-commerce sites, documentation portals, or knowledge bases, a well-designed indexing pipeline directly impacts user satisfaction, conversion rates, and support ticket volume.

Client-Side Indexing with FlexSearch

FlexSearch is a high-performance JavaScript full-text search library designed for browser environments. It operates entirely in memory, uses a novel scoring algorithm, and supports features like partial matching, field-specific search, and result pagination. Let's walk through a complete implementation.

Installation and Setup

npm install flexsearch

Building the Index

The following example indexes a collection of articles, each with a title, body content, and category. We'll create a document index that supports searching across all fields simultaneously while also enabling field-specific queries.

// index-builder.js
import { Document } from 'flexsearch';

// Create a document index with multiple fields
const articleIndex = new Document({
  document: {
    id: 'id',
    index: [
      { field: 'title', tokenize: 'forward', resolution: 3 },
      { field: 'body', tokenize: 'forward', resolution: 5 },
      { field: 'category', tokenize: 'strict' }
    ],
    store: true  // Store original documents for retrieval
  },
  preset: 'match',  // Optimized for relevance matching
  cache: true
});

// Sample article collection
const articles = [
  {
    id: 1,
    title: 'Building Resilient Web APIs with Node.js',
    body: 'This comprehensive guide covers error handling, circuit breakers, and retry strategies for building APIs that withstand failure...',
    category: 'backend'
  },
  {
    id: 2,
    title: 'CSS Container Queries in Practice',
    body: 'Container queries allow components to adapt their layout based on the size of their parent container rather than the viewport...',
    category: 'frontend'
  },
  {
    id: 3,
    title: 'Database Indexing Strategies for High-Performance APIs',
    body: 'Learn about B-tree indexes, partial indexes, and covering indexes to optimize database query performance...',
    category: 'backend'
  },
  {
    id: 4,
    title: 'Implementing Web Content Indexing in Modern Web Applications',
    body: 'A complete walkthrough of client-side and server-side content indexing techniques for building fast search experiences...',
    category: 'fullstack'
  }
];

// Add documents to the index
for (const article of articles) {
  articleIndex.add(article);
}

// Export the index for use in search operations
export { articleIndex };

Performing Searches

// search-engine.js
import { articleIndex } from './index-builder.js';

// Search across all fields
function searchAllFields(query) {
  const results = articleIndex.search(query, {
    limit: 10,
    enrich: true  // Populate results with stored document data
  });
  return results;
}

// Search within a specific field
function searchByField(query, field) {
  const results = articleIndex.search(query, {
    index: field,
    limit: 10,
    enrich: true
  });
  return results;
}

// Combined search with field weighting
function weightedSearch(query) {
  const results = articleIndex.search([
    { field: 'title', query, weight: 3 },
    { field: 'body', query, weight: 1 },
    { field: 'category', query, weight: 2 }
  ], {
    limit: 10,
    enrich: true
  });
  return results;
}

// Example usage
console.log(searchAllFields('indexing'));
// Returns article #3 and #4 with relevance scores

console.log(searchByField('container', 'title'));
// Returns article #2

console.log(weightedSearch('api'));
// Returns articles with "api" in title ranked higher than those with it only in body

Serializing and Loading Prebuilt Indexes

For production use, you'll want to prebuild the index during your build process and ship it as a static asset rather than constructing it on each page load. FlexSearch supports export and import operations.

// build-index-for-production.js
import { articleIndex } from './index-builder.js';
import { writeFileSync } from 'fs';

// Export the index as a serialized object
const exportedData = articleIndex.export();
writeFileSync('./public/search-index.json', JSON.stringify(exportedData));

console.log('Index exported successfully for static serving');

// load-index-in-browser.js
import { Document } from 'flexsearch';

async function loadPrebuiltIndex() {
  const response = await fetch('/search-index.json');
  const exportedData = await response.json();
  
  const articleIndex = new Document({
    document: {
      id: 'id',
      index: [
        { field: 'title', tokenize: 'forward', resolution: 3 },
        { field: 'body', tokenize: 'forward', resolution: 5 },
        { field: 'category', tokenize: 'strict' }
      ],
      store: true
    }
  });
  
  // Import the serialized index data
  articleIndex.import(exportedData);
  
  return articleIndex;
}

// Usage
loadPrebuiltIndex().then(index => {
  const results = index.search('resilient', { enrich: true });
  console.log(results);
});

Server-Side Indexing with Elasticsearch

For applications with large content volumes or complex query requirements, a dedicated search server like Elasticsearch provides advanced capabilities. Elasticsearch builds distributed inverted indexes across clusters, supporting near-real-time indexing, aggregations, and a rich query DSL.

Index Design and Mapping

Before indexing content, define a mapping that specifies how fields should be analyzed. The mapping determines tokenization strategies, which fields are indexed, and how different field types behave.

// elasticsearch-mapping.js
// PUT /articles  (using Elasticsearch REST API)
const mapping = {
  mappings: {
    properties: {
      title: {
        type: 'text',
        analyzer: 'english',
        fields: {
          keyword: { type: 'keyword' },
          // For autocomplete: edge n-gram tokenizer
          completion: { type: 'completion' }
        }
      },
      body: {
        type: 'text',
        analyzer: 'english',
        // Store term positions for phrase queries and highlighting
        term_vector: 'with_positions_offsets'
      },
      category: {
        type: 'keyword',
        // Enable aggregations for faceted search
        aggregatable: true
      },
      author: {
        type: 'keyword'
      },
      publishedDate: {
        type: 'date',
        format: 'yyyy-MM-dd'
      },
      tags: {
        type: 'keyword'
      },
      // For semantic/vector search (Elasticsearch 8.x+)
      embedding: {
        type: 'dense_vector',
        dims: 768,
        index: true,
        similarity: 'cosine'
      }
    }
  },
  settings: {
    number_of_shards: 1,
    number_of_replicas: 1,
    analysis: {
      analyzer: {
        // Custom analyzer for code-heavy content
        code_aware_english: {
          type: 'custom',
          tokenizer: 'standard',
          filter: ['lowercase', 'stop', 'stemmer']
        }
      }
    }
  }
};

// Apply the mapping
async function createIndex(client) {
  const exists = await client.indices.exists({ index: 'articles' });
  if (!exists) {
    await client.indices.create({
      index: 'articles',
      body: mapping
    });
    console.log('Articles index created with custom mapping');
  }
}

Bulk Indexing Content

When indexing large amounts of content, use the bulk API to batch operations together. This dramatically improves throughput compared to indexing documents one at a time.

// bulk-indexer.js
import { Client } from '@elastic/elasticsearch';

const client = new Client({ node: 'http://localhost:9200' });

async function bulkIndexArticles(articles) {
  const operations = [];
  
  for (const article of articles) {
    // Each document requires an action metadata line and a source data line
    operations.push({
      index: {
        _index: 'articles',
        _id: article.id
      }
    });
    operations.push({
      title: article.title,
      body: article.body,
      category: article.category,
      author: article.author,
      publishedDate: article.publishedDate,
      tags: article.tags
    });
  }
  
  // Send all documents in a single bulk request
  const bulkResponse = await client.bulk({
    refresh: true,
    body: operations
  });
  
  // Check for errors
  if (bulkResponse.errors) {
    const erroredItems = bulkResponse.items.filter(
      item => item.index && item.index.error
    );
    console.error(`Failed to index ${erroredItems.length} documents`);
    erroredItems.forEach(item => {
      console.error(`  Document ${item.index._id}: ${item.index.error.reason}`);
    });
  } else {
    console.log(`Successfully indexed ${articles.length} documents`);
  }
  
  return bulkResponse;
}

// Example usage with a larger dataset
const articlesToIndex = [
  {
    id: 'article-001',
    title: 'Understanding Inverted Indexes',
    body: 'An inverted index maps terms to their locations within documents...',
    category: 'search-engines',
    author: 'Jane Doe',
    publishedDate: '2025-01-15',
    tags: ['search', 'indexing', 'theory']
  },
  // ... hundreds more articles
];

bulkIndexArticles(articlesToIndex);

Querying the Index

Elasticsearch's query DSL allows combining full-text search, filters, aggregations, and highlighting in a single request.

// search-service.js
async function searchArticles(client, userQuery, filters = {}) {
  const searchBody = {
    query: {
      bool: {
        must: [
          // Multi-field full-text search with field boosting
          {
            multi_match: {
              query: userQuery,
              fields: ['title^3', 'body^1', 'tags^2'],
              type: 'best_fields',
              fuzziness: 'AUTO',
              operator: 'and'
            }
          }
        ],
        filter: [
          // Apply structured filters without affecting relevance scoring
          filters.category && {
            term: { category: filters.category }
          },
          filters.author && {
            term: { author: filters.author }
          },
          filters.dateRange && {
            range: {
              publishedDate: {
                gte: filters.dateRange.start,
                lte: filters.dateRange.end
              }
            }
          }
        ].filter(Boolean)
      }
    },
    // Highlight matching terms in results
    highlight: {
      fields: {
        title: { number_of_fragments: 0 },
        body: { fragment_size: 150, number_of_fragments: 3 }
      },
      pre_tags: [''],
      post_tags: ['']
    },
    // Faceted aggregations for sidebar filters
    aggs: {
      categories: {
        terms: { field: 'category', size: 20 }
      },
      authors: {
        terms: { field: 'author', size: 20 }
      },
      top_tags: {
        terms: { field: 'tags', size: 30 }
      }
    },
    // Pagination
    from: (filters.page - 1) * filters.pageSize || 0,
    size: filters.pageSize || 20
  };
  
  const response = await client.search({
    index: 'articles',
    body: searchBody
  });
  
  return {
    total: response.hits.total.value,
    results: response.hits.hits.map(hit => ({
      id: hit._id,
      score: hit._score,
      title: hit._source.title,
      body: hit._source.body,
      category: hit._source.category,
      author: hit._source.author,
      publishedDate: hit._source.publishedDate,
      highlights: hit.highlight || {}
    })),
    aggregations: response.aggregations,
    page: filters.page || 1,
    pageSize: filters.pageSize || 20
  };
}

// Example query
const results = await searchArticles(client, 'indexing strategies', {
  category: 'search-engines',
  page: 1,
  pageSize: 10
});

Hybrid Indexing with SQLite FTS5

Not every application needs a distributed search cluster. SQLite's FTS5 (Full-Text Search v5) module provides a remarkably capable embedded indexing engine—perfect for single-server applications, desktop apps built with Electron, or edge-deployed services. It supports prefix queries, phrase queries, NEAR operators, and highlighted snippet extraction.

-- schema.sql
-- Create the main content table
CREATE TABLE documents (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  title TEXT NOT NULL,
  body TEXT NOT NULL,
  category TEXT,
  author TEXT,
  created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);

-- Create a virtual FTS5 table that indexes title and body
-- The 'content' column combines both fields for unified search
CREATE VIRTUAL TABLE documents_fts USING fts5(
  title,
  body,
  category UNINDEXED,  -- Stored but not tokenized (used for filtering)
  content='documents',
  content_rowid='id'
);

-- Triggers keep the FTS index synchronized with the main table
CREATE TRIGGER documents_fts_insert AFTER INSERT ON documents BEGIN
  INSERT INTO documents_fts(rowid, title, body, category)
  VALUES (new.id, new.title, new.body, new.category);
END;

CREATE TRIGGER documents_fts_update AFTER UPDATE ON documents BEGIN
  UPDATE documents_fts SET
    title = new.title,
    body = new.body,
    category = new.category
  WHERE rowid = old.id;
END;

CREATE TRIGGER documents_fts_delete AFTER DELETE ON documents BEGIN
  DELETE FROM documents_fts WHERE rowid = old.id;
END;

// sqlite-search.js
import Database from 'better-sqlite3';
import { join } from 'path';

const db = new Database(join(process.cwd(), 'content.db'));
db.pragma('journal_mode = WAL');
db.pragma('foreign_keys = ON');

// Execute the schema
db.exec(`
  CREATE TABLE IF NOT EXISTS documents (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    title TEXT NOT NULL,
    body TEXT NOT NULL,
    category TEXT,
    author TEXT,
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP
  );
  
  CREATE VIRTUAL TABLE IF NOT EXISTS documents_fts USING fts5(
    title,
    body,
    category UNINDEXED,
    content='documents',
    content_rowid='id'
  );
`);

// Insert a document (triggers handle FTS synchronization)
const insertDocument = db.prepare(`
  INSERT INTO documents (title, body, category, author)
  VALUES (?, ?, ?, ?)
`);

// Full-text search with snippet highlighting
const searchDocuments = db.prepare(`
  SELECT
    d.id,
    d.title,
    d.category,
    d.author,
    d.created_at,
    highlight(documents_fts, 0, '', '') AS title_highlight,
    snippet(documents_fts, 1, '', '', '...', 64) AS body_snippet,
    rank AS relevance
  FROM documents_fts
  JOIN documents d ON documents_fts.rowid = d.id
  WHERE documents_fts MATCH ?
    ${''/* Optional: filter by category */''}
  ORDER BY rank
  LIMIT ?
  OFFSET ?
`);

// Insert sample content
const sampleDocs = [
  ['Inverted Index Explained', 'An inverted index is a database index structure that maps content...', 'tutorial', 'Alice'],
  ['Building Search Features', 'Implementing search requires understanding tokenization, stemming, and relevance scoring...', 'tutorial', 'Bob'],
  ['SQLite FTS5 Deep Dive', 'The FTS5 module provides full-text search capabilities embedded directly within SQLite...', 'reference', 'Alice']
];

const insertMany = db.transaction(() => {
  for (const doc of sampleDocs) {
    insertDocument.run(...doc);
  }
});
insertMany();

// Search function
function search(query, limit = 10, offset = 0) {
  // FTS5 supports boolean operators, phrase queries, and prefix queries
  // Example: 'index* AND explain*' matches both 'index' and 'indexing'
  const results = searchDocuments.all(query, limit, offset);
  
  return results.map(row => ({
    id: row.id,
    title: row.title,
    highlightedTitle: row.title_highlight,
    snippet: row.body_snippet,
    category: row.category,
    author: row.author,
    relevance: row.relevance
  }));
}

// Example queries demonstrating FTS5 query syntax
console.log(search('inverted index'));         // Phrase search
console.log(search('index* AND structur*'));   // Prefix + boolean
console.log(search('NEAR(index explain, 3)')); // Proximity search

Vector Indexing for Semantic Search

Traditional keyword indexes match tokens literally. Vector indexes, by contrast, encode semantic meaning into dense numerical embeddings, allowing searches like "ways to make my app faster" to find documents about performance optimization even when they never contain the exact query words. This approach uses embedding models to convert text into high-dimensional vectors, then performs nearest-neighbor search in that vector space.

// vector-indexer.js
// This example uses Transformers.js for in-browser embedding generation
// and a simple cosine-similarity index stored in IndexedDB

import { pipeline } from '@xenova/transformers';
import { openDB } from 'idb';

// Initialize the embedding pipeline (runs entirely in the browser)
let embedder = null;
async function getEmbedder() {
  if (!embedder) {
    // Load a distilled model optimized for browser use
    embedder = await pipeline(
      'feature-extraction',
      'Xenova/all-MiniLM-L6-v2'
    );
  }
  return embedder;
}

// Generate embedding vector for a text
async function embedText(text) {
  const model = await getEmbedder();
  const output = await model(text, { pooling: 'mean', normalize: true });
  // Returns a Float32Array of 384 dimensions
  return Array.from(output.data);
}

// Store vectors in IndexedDB
const dbPromise = openDB('vector-index', 1, {
  upgrade(db) {
    db.createObjectStore('vectors', { keyPath: 'id' });
    db.createObjectStore('metadata', { keyPath: 'id' });
  }
});

// Index a document: embed its content and store the vector
async function indexDocument(id, title, body) {
  const combinedText = `${title}\n${body}`;
  const embedding = await embedText(combinedText);
  
  const db = await dbPromise;
  const tx = db.transaction(['vectors', 'metadata'], 'readwrite');
  
  await tx.objectStore('vectors').put({
    id,
    embedding,
    // Store magnitude for faster cosine similarity computation
    magnitude: Math.sqrt(embedding.reduce((sum, v) => sum + v * v, 0))
  });
  
  await tx.objectStore('metadata').put({
    id,
    title,
    body,
    indexedAt: new Date().toISOString()
  });
  
  await tx.done;
}

// Cosine similarity between two vectors
function cosineSimilarity(vecA, vecB, magA, magB) {
  let dotProduct = 0;
  for (let i = 0; i < vecA.length; i++) {
    dotProduct += vecA[i] * vecB[i];
  }
  return dotProduct / (magA * magB);
}

// Search for similar documents
async function semanticSearch(query, topK = 5) {
  const queryEmbedding = await embedText(query);
  const queryMagnitude = Math.sqrt(
    queryEmbedding.reduce((sum, v) => sum + v * v, 0)
  );
  
  const db = await dbPromise;
  const allVectors = await db.getAll('vectors');
  
  // Compute similarity scores
  const scored = allVectors.map(entry => ({
    id: entry.id,
    score: cosineSimilarity(queryEmbedding, entry.embedding, queryMagnitude, entry.magnitude)
  }));
  
  // Sort by descending similarity and take top K
  scored.sort((a, b) => b.score - a.score);
  const topResults = scored.slice(0, topK);
  
  // Fetch metadata for top results
  const results = [];
  for (const result of topResults) {
    const metadata = await db.get('metadata', result.id);
    results.push({
      id: result.id,
      title: metadata.title,
      body: metadata.body,
      similarity: Math.round(result.score * 1000) / 1000
    });
  }
  
  return results;
}

// Example usage
async function demo() {
  await indexDocument('doc-1', 'Optimizing React Rendering', 
    'Learn how to use memo, useMemo, and useCallback to reduce unnecessary re-renders...');
  await indexDocument('doc-2', 'Database Query Performance',
    'Techniques for speeding up SQL queries including proper indexing and query planning...');
  await indexDocument('doc-3', 'CSS Animation Performance',
    'Use transform and opacity for smooth 60fps animations without layout thrashing...');
  
  // This will find all performance-related documents even though
  // the query words differ from the document titles
  const results = await semanticSearch('making things run faster', 2);
  console.log(results);
  // Returns documents about optimization, performance, and speed
  // ranked by conceptual similarity rather than keyword overlap
}

Building a Complete Search Pipeline

A production-grade indexing system requires coordination between content ingestion, index updates, and query serving. Here's a comprehensive pipeline that handles content changes gracefully, supports incremental updates, and provides a clean API for consumers.

// search-pipeline.js
// A unified search pipeline coordinating content updates and queries

class SearchPipeline {
  constructor(options = {}) {
    this.indexName = options.indexName || 'content';
    this.batchSize = options.batchSize || 100;
    this.indexQueue = [];
    this.processingTimer = null;
    this.indexVersion = 0;
    this.listeners = new Map();
  }
  
  // Register a content source (database, CMS, file system)
  registerSource(source) {
    source.on('create', (doc) => this.enqueue('index', doc));
    source.on('update', (doc) => this.enqueue('reindex', doc));
    source.on('delete', (id) => this.enqueue('remove', id));
    return this;
  }
  
  // Enqueue an indexing operation with debounced batch processing
  enqueue(operation, payload) {
    this.indexQueue.push({ operation, payload, timestamp: Date.now() });
    
    // Process after a short debounce to batch multiple rapid changes
    clearTimeout(this.processingTimer);
    this.processingTimer = setTimeout(() => this.processBatch(), 200);
  }
  
  async processBatch() {
    if (this.indexQueue.length === 0) return;
    
    const batch = this.indexQueue.splice(0, this.batchSize);
    const operationsByType = {
      index: [],
      reindex: [],
      remove: []
    };
    
    for (const item of batch) {
      operationsByType[item.operation].push(item.payload);
    }
    
    const results = {
      indexed: 0,
      reindexed: 0,
      removed: 0,
      errors: []
    };
    
    // Process removals first to avoid stale content
    for (const id of operationsByType.remove) {
      try {
        await this.removeFromIndex(id);
        results.removed++;
      } catch (err) {
        results.errors.push({ operation: 'remove', id, error: err.message });
      }
    }
    
    // Process new content
    for (const doc of operationsByType.index) {
      try {
        await this.addToIndex(doc);
        results.indexed++;
      } catch (err) {
        results.errors.push({ operation: 'index', doc, error: err.message });
      }
    }
    
    // Process updates (remove old version, add new)
    for (const doc of operationsByType.reindex) {
      try {
        await this.removeFromIndex(doc.id);
        await this.addToIndex(doc);
        results.reindexed++;
      } catch (err) {
        results.errors.push({ operation: 'reindex', doc, error: err.message });
      }
    }
    
    this.indexVersion++;
    this.notifyListeners('batchComplete', results);
    
    // Process remaining queue items if any
    if (this.indexQueue.length > 0) {
      this.processingTimer = setTimeout(() => this.processBatch(), 100);
    }
    
    return results;
  }
  
  async addToIndex(doc) {
    // Normalize and tokenize content
    const tokens = this.tokenize(doc.content);
    const enriched = {
      ...doc,
      tokens,
      termFrequency: this.computeTermFrequency(tokens),
      indexedAt: new Date().toISOString()
    };
    
    // Store in your chosen index backend (FlexSearch, Elasticsearch, SQLite, etc.)
    await this.indexBackend.add(enriched);
  }
  
  async removeFromIndex(id) {
    await this.indexBackend.remove(id);
  }
  
  tokenize(text) {
    return text
      .toLowerCase()
      .replace(/[^a-z0-9\s-]/g, '')
      .split(/\s+/)
      .filter(token => token.length > 1 && !this.isStopWord(token))
      .map(token => this.stem(token));
  }
  
  isStopWord(word) {
    const stopWords = new Set([
      'the', 'and', 'is', 'in', 'it', 'of', 'to', 'a', 'an', 'for',
      'on', 'with', 'as', 'at', 'by', 'or', 'be', 'was', 'are'
    ]);
    return stopWords.has(word);
  }
  
  stem(word) {
    // Simple Porter-inspired stemming
    if (word.endsWith('ing')) return word.slice(0, -3);
    if (word.endsWith('s') && !word.endsWith('ss')) return word.slice(0, -1);
    if (word.endsWith('ed')) return word.slice(0, -2);
    return word;
  }
  
  computeTermFrequency(tokens) {
    const freq = {};
    for (const token of tokens) {
      freq[token] = (freq[token] || 0) + 1;
    }
    return freq;
  }
  
  notifyListeners(event, data) {
    const callbacks = this.listeners.get(event) || [];
    callbacks.forEach(cb => cb(data));
  }
  
  on(event, callback) {
    if (!this.listeners.has(event)) {
      this.listeners.set(event, []);
    }
    this.listeners.get(event).push(callback);
    return this;
  }
  
  // Force immediate processing of all queued operations
  async flush() {
    clearTimeout(this.processingTimer);
    while (this.indexQueue.length > 0) {
      await this.processBatch();
    }
  }
  
  // Get current index statistics
  async stats() {
    return {
      indexName: this.indexName,
      version: this.indexVersion,
      queueLength: this.indexQueue.length,
      documentCount: await this.indexBackend.count(),
      lastProcessed: this.lastBatchTime || null
    };
  }
}

// Usage example
const pipeline = new SearchPipeline({
  indexName: 'articles',
  batchSize: 50
});

pipeline.on('batchComplete', (results) => {
  console.log(`Indexed: ${results.indexed}, Reindexed: ${results.reindexed}, Removed: ${results.removed}`);
  if (results.errors.length > 0) {
    console.warn(`Errors: ${results.errors.length}`);
  }
});

// Connect to content sources
// pipeline.registerSource(databaseChangeStream);
// pipeline.registerSource(cmsWebhookFeed);

Best Practices for Production Indexing

1. Choose the Right Tokenization Strategy

Tokenization decisions ripple through every aspect of search quality. For English content, use stemming or lemmatization to normalize word forms. For multilingual applications, employ language-specific analyzers. For code or technical content, consider custom tokenizers that preserve symbols like underscores and dots. Always test tokenization output against real user queries—what looks correct in theory often produces surprising results in practice.

// Demonstrating tokenization impact
const badTokenizer = (text) => text.split(/\s+/);
const goodTokenizer = (text) => text
  .toLowerCase()
  .replace(/[^a-z0-9\s-]/g, '')
  .split(/\s+/)
  .filter(t => t.length > 1);

// "Running shoes are GREAT for marathon-training!"
// Bad:  ["Running", "shoes", "are", "GREAT", "for", "marathon-training!"]
// Good: ["running", "shoes", "great", "marathon", "training"]
// A query for "run shoe" now matches the good tokens via stemming

2. Implement Incremental Indexing

Full reindexing of large content sets is expensive. Design your pipeline to handle incremental updates: when a document changes, remove its old tokens and insert new ones rather than rebuilding the entire index. Maintain a version identifier or timestamp to detect stale entries. For SQLite FTS5, the triggers shown earlier handle this automatically. For Elasticsearch, use the update API with doc_as_upsert. For client-side indexes, implement a diff-based approach.

3. Monitor Index Health

Track index size, indexing latency, and query performance over time. An index that grows without bound slows queries and consumes excessive memory. Implement retention policies for ephemeral content. Set up alerts for indexing failures—a silently broken index is worse than no index at all because users trust results that may be incomplete.

4. Secure Your Index

Indexes often contain the full text of potentially sensitive documents. Apply the same access control logic to search results that you apply to direct document retrieval. Never expose internal document IDs or unredacted content through search APIs. For client-side indexes, be mindful of what you ship to the browser—prebuilt indexes embedded in static assets are readable by anyone who downloads them.

5. Combine Full-Text and Structured Filters Efficiently

When a query includes both text search and filters (category, date range), apply filters after the full-text scoring to avoid the "filter-then-search" pitfall where relevant documents are excluded before ranking. Modern engines handle this with filter contexts that skip scoring on filtered clauses. Always measure whether your filter application order affects result quality.

6. Cache Aggressively, Invalidate Precisely

Search results for common queries can be cached at multiple levels: CDN edge caches for static result sets, in-memory caches for frequent queries, and even precomputed result pages for high-traffic terms. The challenge is cache invalidation—when content changes, only the affected query results should be purged. Implement query-result dependency tracking: record which document IDs contribute to each cached query and invalidate only those entries when documents change.

7. Design for Index Portability

Build your indexing pipeline so you can switch backends without rewriting your application. Abstract the index operations behind a clean interface. This protects against vendor changes and allows you to move from a simple in-process index to a distributed search cluster as your scale grows—without redesigning your entire search architecture.

// Index abstraction interface
class IndexBackend {
  async add(document) { throw new Error('Not implemented'); }
  async remove(id) { throw new Error('Not implemented'); }
  async search(query, options) { throw new Error('Not implemented'); }
  async count() { throw new Error('Not implemented'); }
  async export

🚀 Need a reliable AI agent for your project?

Deploy Hermes Agent in 10 minutes. Managed hosting, zero DevOps.

Get Started — $23.99/mo
← Back to all articles