MongoDB Application Performance: Profiling and Optimization

Understanding MongoDB Performance Profiling

MongoDB performance profiling is the systematic process of monitoring, recording, and analyzing database operations to identify slow-running queries, inefficient patterns, and resource bottlenecks. At its core, profiling gives developers visibility into exactly how MongoDB processes each operation — revealing execution times, document counts examined versus returned, index usage, and query plans. This visibility is essential because without it, performance issues remain hidden until they manifest as degraded user experience or system failures under load.

The MongoDB profiler operates at the database level, capturing detailed metadata about every operation that exceeds configurable thresholds. It stores this data in a special capped collection called system.profile, which developers can query just like any other collection. Profiling data includes the full query shape, execution time in milliseconds, the number of documents scanned, the number returned, whether an index was used, and even the specific keys examined. This granular detail transforms guesswork into precise, data-driven optimization.

Why Performance Profiling Matters

Unoptimized queries are often the root cause of poor application performance. A query that scans 100,000 documents to return only 10 results wastes memory, CPU cycles, and I/O bandwidth. As data grows, these inefficiencies compound non-linearly — what runs in 10ms with 10,000 documents may take 500ms with a million documents. Profiling catches these problems early, before they become production incidents. Additionally, profiling helps you understand whether your indexes are being used correctly, whether you have missing indexes, or whether your data model itself needs restructuring.

Beyond individual query performance, profiling reveals systemic patterns: hot collections receiving disproportionate traffic, operations that block others due to locks (in earlier versions or specific configurations), and aggregation pipelines that consume excessive memory. In microservice architectures where multiple services share a MongoDB cluster, profiling helps attribute resource consumption to specific services and endpoints, facilitating cross-team accountability.

Enabling and Configuring the MongoDB Profiler

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

The MongoDB profiler is controlled through database-level settings. It supports three distinct profiling levels, each offering a different balance of data granularity and overhead. Configuration is performed using the db.setProfilingLevel() command in mongosh or via driver-specific APIs.

Profiling Levels Explained

Level 0: The profiler is off. No data is collected. This is the default setting.
Level 1: The profiler collects data only for slow operations — those exceeding a specified threshold in milliseconds. This is the recommended setting for production environments because it limits overhead while capturing the most impactful queries.
Level 2: The profiler collects data for all operations, regardless of execution time. This provides comprehensive visibility but introduces measurable overhead. It is suitable for development and staging environments, or for short diagnostic windows in production under careful supervision.

Enabling the Profiler via mongosh

To enable profiling, connect to your mongod instance using mongosh and run the following command against the target database. The second argument specifies the slow query threshold in milliseconds.

// Connect to the target database
use myapp_database;

// Enable profiling level 1 with a 100ms slow query threshold
db.setProfilingLevel(1, { slowms: 100 });

// Verify the current profiling settings
db.getProfilingStatus();

The getProfilingStatus() output looks like this:

{
  "was": 0,
  "slowms": 100,
  "sampleRate": 1,
  "ok": 1
}

Here, was indicates the previous profiling level, slowms shows the current threshold, and sampleRate controls sampling (1 means capture every qualifying operation).

Setting Slow Query Thresholds Strategically

The slowms threshold should be tuned to your application's performance expectations. For a real-time API serving users, you might set it to 50ms. For a background analytics job, 500ms might be more appropriate. Setting the threshold too low floods the system.profile collection with noise; setting it too high misses opportunities for optimization. A pragmatic starting point is 100ms, then adjust based on initial findings.

Using Sampling to Reduce Overhead

For high-throughput production systems, profiling every slow operation can still generate significant overhead. MongoDB supports a sampling rate option that profiles only a fraction of slow queries. This is configured using the sampleRate parameter:

// Profile only 20% of slow operations
db.setProfilingLevel(1, { slowms: 100, sampleRate: 0.2 });

A sampleRate of 0.2 means approximately one in five qualifying slow operations will be logged. This reduces profiler overhead while still providing statistically representative data for analysis.

Analyzing Profiler Data

All profiling data is stored in the system.profile collection within the profiled database. This is a capped collection, meaning it has a fixed size and older documents are automatically evicted as new ones arrive. The default size is typically adequate for short-term analysis, but you may need to increase it for longer diagnostic windows.

Querying the system.profile Collection

You can query system.profile like any other collection, filtering by operation type, namespace, execution time, or any other recorded field. Here are practical query examples:

// Find the top 10 slowest queries in the last hour
db.system.profile
  .find({ ts: { $gte: new Date(Date.now() - 3600000) } })
  .sort({ millis: -1 })
  .limit(10)
  .pretty();

// Find all queries targeting a specific collection
db.system.profile
  .find({ ns: 'myapp_database.users' })
  .sort({ millis: -1 })
  .limit(20);

// Find queries that performed a full collection scan (no index used)
db.system.profile
  .find({ 'planSummary': { $regex: /COLLSCAN/ } })
  .sort({ millis: -1 })
  .limit(20);

// Find operations slower than 500ms
db.system.profile
  .find({ millis: { $gt: 500 } })
  .sort({ millis: -1 });

Understanding Profiler Document Structure

Each document in system.profile contains rich metadata. Here is a typical entry for a slow query, with annotations explaining key fields:

{
  "op": "query",                    // Operation type: query, update, insert, remove, command
  "ns": "mydb.orders",              // Namespace: database.collection
  "command": {                      // The actual command executed
    "find": "orders",
    "filter": { "status": "pending", "customer_id": "cust_1234" },
    "sort": { "created_at": -1 },
    "limit": 50
  },
  "keysExamined": 0,                // Index keys examined — 0 indicates no index used
  "docsExamined": 850000,           // Documents scanned to fulfill the query
  "nreturned": 50,                  // Documents returned to the client
  "millis": 1240,                   // Execution time in milliseconds
  "planSummary": "COLLSCAN",        // Query plan summary
  "execStats": {                    // Detailed execution statistics
    "stage": "COLLSCAN",
    "nReturned": 50,
    "executionTimeMillisEstimate": 1200
  },
  "ts": ISODate("2025-01-15T14:32:10.123Z"),
  "client": "192.168.1.100:54321",
  "user": "app_user@admin"
}

The ratio of docsExamined to nreturned is a critical efficiency metric. In the example above, 850,000 documents were scanned to return just 50 — a ratio of 17,000:1. An efficient indexed query would have a ratio close to 1:1. The planSummary field shows "COLLSCAN" (collection scan), confirming no index was used, which explains the poor performance.

Using explain() for Query Analysis

The explain() method provides on-demand query plan analysis without enabling the profiler. It shows exactly how MongoDB will execute a query, including which indexes are considered, which index is chosen, and estimated execution statistics. This is invaluable during development for validating that indexes are used as expected.

Explain Verbosity Modes

The explain() method supports three verbosity modes via the explain method or by passing a verbosity string:

queryPlanner (default): Shows the query plan MongoDB would choose, including rejected plans. No execution statistics.
executionStats: Includes the winning plan plus actual execution statistics like documents examined and execution time.
allPlansExecution: Shows execution statistics for both the winning plan and all rejected candidate plans. Useful for understanding why MongoDB chose a particular index.

// Basic explain with queryPlanner mode (default)
db.orders.explain().find({
  status: "pending",
  customer_id: "cust_1234"
}).sort({ created_at: -1 }).limit(50);

// Explain with execution statistics
db.orders.explain("executionStats").find({
  status: "pending",
  customer_id: "cust_1234"
}).sort({ created_at: -1 }).limit(50);

// Full plan analysis with all candidate plans
db.orders.explain("allPlansExecution").find({
  status: "pending",
  customer_id: "cust_1234"
}).sort({ created_at: -1 }).limit(50);

Interpreting Explain Output

Let's examine a complete executionStats output for a query that uses an index efficiently:

{
  "queryPlanner": {
    "namespace": "mydb.orders",
    "indexFilterSet": false,
    "winningPlan": {
      "stage": "LIMIT",
      "limitAmount": 50,
      "inputStage": {
        "stage": "SORT",
        "sortPattern": { "created_at": -1 },
        "inputStage": {
          "stage": "IXSCAN",
          "indexName": "status_customer_created_idx",
          "keysExamined": 52,
          "direction": "forward"
        }
      }
    },
    "rejectedPlans": [
      {
        "stage": "SORT",
        "inputStage": {
          "stage": "IXSCAN",
          "indexName": "status_idx",
          "keysExamined": 25000
        }
      }
    ]
  },
  "executionStats": {
    "executionSuccess": true,
    "nReturned": 50,
    "executionTimeMillis": 8,
    "totalKeysExamined": 52,
    "totalDocsExamined": 50,
    "executionStages": {
      "stage": "LIMIT",
      "nReturned": 50,
      "executionTimeMillisEstimate": 6,
      "inputStages": [
        {
          "stage": "SORT",
          "nReturned": 50,
          "executionTimeMillisEstimate": 5,
          "inputStages": [
            {
              "stage": "IXSCAN",
              "nReturned": 52,
              "keysExamined": 52,
              "docsExamined": 50,
              "indexName": "status_customer_created_idx"
            }
          ]
        }
      ]
    }
  }
}

Key observations: The winning plan uses the compound index status_customer_created_idx, examining only 52 index keys and 50 documents to return 50 results — a nearly perfect 1:1 ratio. Execution time is just 8ms. The rejected plan using status_idx alone would have scanned 25,000 keys, demonstrating why MongoDB's query planner selected the more selective index.

Index Optimization Strategies

Indexes are the single most powerful lever for MongoDB performance optimization. A well-designed index transforms a full collection scan into a highly efficient lookup. However, indexes also consume memory and impose write overhead, so they must be chosen judiciously.

Creating Indexes That Match Query Patterns

The fundamental principle of index design is the ESR rule: Equality, Sort, Range. Fields used in equality filters (exact matches) should appear first in the index definition. Fields used for sorting come next. Fields used for range queries (like $gt, $lt, or $in) come last. This ordering allows MongoDB to use the index for both filtering and sorting without an in-memory sort stage.

// For query: db.orders.find({ status: "pending", customer_id: "cust_1234" })
//            .sort({ created_at: -1 }).limit(50)
// ESR rule: equality on status and customer_id, sort on created_at
db.orders.createIndex(
  { status: 1, customer_id: 1, created_at: -1 },
  { name: "status_customer_created_idx" }
);

Covered Queries

A covered query is one where all fields needed by the query — both the filter fields and the projected fields — are present in the index. MongoDB can satisfy a covered query entirely from the index without ever looking at the actual documents, dramatically reducing I/O. To achieve this, create an index that includes all projected fields as the last components.

// Query: db.orders.find(
//   { status: "pending" },
//   { _id: 0, status: 1, customer_id: 1, total: 1 }
// ).sort({ created_at: -1 })
// All fields (status, customer_id, total, created_at) should be in the index
db.orders.createIndex(
  { status: 1, created_at: -1, customer_id: 1, total: 1 },
  { name: "covered_status_sort_idx" }
);

// Verify coverage with explain
db.orders.explain("executionStats").find(
  { status: "pending" },
  { _id: 0, status: 1, customer_id: 1, total: 1 }
).sort({ created_at: -1 });

// Look for "totalDocsExamined": 0 in the output — this confirms a covered query

Index Selectivity and Cardinality

An index is most effective when its leading fields have high cardinality — meaning many distinct values. An index on a boolean field with only two values is less selective than an index on a unique user ID. Use the following aggregation to assess field cardinality before creating an index:

// Assess cardinality of a potential index field
db.orders.aggregate([
  { $group: { _id: "$customer_id", count: { $sum: 1 } } },
  { $count: "distinct_values" }
]);

// For compound index candidate, check combined selectivity
db.orders.aggregate([
  { $group: { _id: { status: "$status", customer_id: "$customer_id" }, count: { $sum: 1 } } },
  { $count: "distinct_combinations" }
]);

Identifying and Removing Unused Indexes

Unused indexes waste memory and slow down writes. MongoDB tracks index usage statistics, which you can query to find indexes that are never accessed:

// Find all indexes for a collection with usage statistics
db.orders.aggregate([
  { $indexStats: {} }
]).forEach(function(doc) {
  printjson({
    name: doc.name,
    key: doc.key,
    accesses: doc.accesses.ops,
    since: doc.accesses.since
  });
});

// Look for indexes with zero accesses over a long period
// These are candidates for removal
db.orders.dropIndex("unused_index_name");

Common Performance Bottlenecks and Solutions

Missing Indexes Leading to Collection Scans

The most common bottleneck is queries running without suitable indexes, resulting in COLLSCAN in the query plan. These queries scan every document in the collection, causing linear degradation as data grows. The fix is straightforward: identify the query pattern from profiler data and create an appropriate index following the ESR rule.

// Identify collection scans from profiler
db.system.profile.find({
  'planSummary': { $regex: /COLLSCAN/ },
  'op': 'query'
}).sort({ millis: -1 }).limit(20);

// Extract the query pattern and create a matching index
// For example, if the query filters on { category: "electronics" }
db.products.createIndex({ category: 1, price: -1 });

Large Skip Values on Sorted Queries

Queries using skip() with large values force MongoDB to traverse and discard many documents. For pagination, prefer range-based pagination using indexed fields over offset-based skip approaches.

// Inefficient: large skip value
db.orders.find({ status: "pending" })
  .sort({ _id: 1 })
  .skip(100000)
  .limit(50);

// Efficient: range-based pagination using last seen _id
const lastSeenId = ObjectId("64a1b2c3...");
db.orders.find({
  status: "pending",
  _id: { $gt: lastSeenId }
}).sort({ _id: 1 }).limit(50);

Aggregation Pipeline Memory Issues

Aggregation stages like $group, $sort, and $lookup can consume substantial memory. When memory exceeds the allowable limit, MongoDB spills to disk, causing severe slowdowns. Use the allowDiskUse option and optimize pipeline order — place $match and $limit stages early to reduce document count flowing through subsequent stages.

// Optimized aggregation: $match early, $sort with index
db.orders.aggregate([
  { $match: { status: "pending", created_at: { $gte: startDate } } },
  { $sort: { created_at: -1 } },
  { $limit: 1000 },
  { $group: { _id: "$customer_id", total_spent: { $sum: "$amount" } } },
  { $sort: { total_spent: -1 } }
], { allowDiskUse: true });

// Ensure an index supports the $match and $sort stages
db.orders.createIndex({ status: 1, created_at: -1 });

Write Contention and Locking

In deployments with high write volumes, contention can arise from multiple updates targeting the same document or from index maintenance during writes. Use the profiler to identify hot collections and consider sharding to distribute write load across shards. For update-heavy workloads, review whether all existing indexes are justified, as each index imposes a write penalty.

Large Document Sizes and Unbounded Arrays

Documents that grow without bound — particularly arrays that accumulate elements indefinitely — cause performance degradation as MongoDB must read and write increasingly large documents. The profiler reveals this through increasing docsExamined times for queries on such collections. Consider restructuring the data model to use separate collections for the growing dimension (a "buckets" pattern or one-to-many relationship) instead of embedding unbounded arrays.

Best Practices for MongoDB Performance Optimization

Establish a Performance Baseline

Before optimizing, capture baseline performance metrics: average query latency, throughput, and resource utilization under representative load. This allows you to measure the impact of each optimization quantitatively. Use the profiler with a reasonable slowms threshold and periodically export system.profile data for trend analysis.

Index Proactively, Not Excessively

Create indexes for queries you know your application will run, based on the ESR rule. Avoid creating indexes speculatively — each index adds write overhead and memory pressure. Regularly audit index usage using $indexStats and remove indexes with zero or negligible access counts over extended periods.

Use explain() During Development

Make explain("executionStats") a mandatory step in your development workflow for any new query. Check that docsExamined is close to nReturned, that the winning plan uses an index, and that rejected plans are genuinely inferior. This catches performance issues before they reach production.

Monitor Key Metrics Continuously

Beyond the profiler, monitor host-level metrics (CPU, memory, disk I/O) and MongoDB-specific metrics (operation counters, connection pools, replication lag). Tools like MongoDB Atlas monitoring, Prometheus with MongoDB exporters, or MongoDB Ops Manager provide dashboards that complement profiler data with system-level context.

Optimize Data Modeling for Access Patterns

MongoDB's document model gives flexibility, but performance optimization requires designing schemas aligned with how data is accessed. If you frequently query orders by customer, consider embedding order summaries in customer documents or maintaining a separate denormalized collection optimized for those reads. Use the profiler to validate that your data model serves your dominant access patterns efficiently.

Batch Operations and Use Bulk Writes

For bulk inserts, updates, or deletes, use bulkWrite() operations to reduce network round trips and server overhead. The profiler will show these as single commands with sub-operations, making them easier to analyze than hundreds of individual operation entries.

// Efficient bulk write example
db.orders.bulkWrite([
  { insertOne: { document: { customer_id: "cust_100", amount: 250 } } },
  { updateOne: { 
      filter: { _id: orderId1 }, 
      update: { $set: { status: "shipped" } } 
  }},
  { updateOne: { 
      filter: { _id: orderId2 }, 
      update: { $set: { status: "shipped" } } 
  }},
  { deleteOne: { filter: { _id: staleOrderId } } }
], { ordered: false });

Leverage Query Caching and Result Set Reuse

MongoDB does not have a built-in query result cache, but you can implement caching at the application layer using Redis or in-memory caches for frequently accessed, relatively static query results. The profiler helps identify cache-worthy queries — those that run frequently with identical filters and return the same results. Focus caching efforts on queries with high execution counts and moderate latency in system.profile.

Putting It All Together: A Practical Optimization Workflow

Let's walk through a complete optimization workflow using the tools and techniques covered above. This simulates a real-world scenario where an e-commerce application experiences slow order listing performance.

// Step 1: Enable the profiler in the development environment
use ecommerce_db;
db.setProfilingLevel(1, { slowms: 50 });

// Step 2: Run the problematic query to capture it in the profiler
db.orders.find({
  customer_id: "cust_abc123",
  status: { $in: ["pending", "processing"] },
  created_at: { $gte: new Date("2024-01-01") }
}).sort({ created_at: -1 }).limit(25);

// Step 3: Query system.profile to see what happened
db.system.profile.find({ "ns": "ecommerce_db.orders" })
  .sort({ ts: -1 }).limit(1).pretty();

// Output reveals: millis: 450, docsExamined: 200000, nreturned: 25
// planSummary: COLLSCAN — clearly needs an index

// Step 4: Use explain() to validate the query plan before creating an index
db.orders.explain("executionStats").find({
  customer_id: "cust_abc123",
  status: { $in: ["pending", "processing"] },
  created_at: { $gte: new Date("2024-01-01") }
}).sort({ created_at: -1 }).limit(25);

// Step 5: Create an index following ESR rule
// Equality: customer_id (exact match)
// Equality/Sort: status (exact with $in, then sort on created_at)
// Range: created_at (already used in sort, so index order matters)
db.orders.createIndex({
  customer_id: 1,
  status: 1,
  created_at: -1
}, { name: "cust_status_created_idx" });

// Step 6: Verify improvement with explain
db.orders.explain("executionStats").find({
  customer_id: "cust_abc123",
  status: { $in: ["pending", "processing"] },
  created_at: { $gte: new Date("2024-01-01") }
}).sort({ created_at: -1 }).limit(25);

// Now shows: totalKeysExamined: 25, totalDocsExamined: 25, executionTimeMillis: 3
// A dramatic improvement from 200,000 documents scanned to just 25

// Step 7: Monitor ongoing performance
// Check that the new index is being used actively
db.orders.aggregate([{ $indexStats: {} }]);

This workflow — profile, identify slow queries, explain to understand query plans, create targeted indexes, verify with explain, and monitor — should become second nature. Each step is backed by concrete data, eliminating guesswork and ensuring that optimizations actually solve the observed problem.

Advanced Profiling Techniques

Custom Profiler Data Analysis with Aggregation Pipeline

For sophisticated analysis, use MongoDB's aggregation pipeline on the system.profile collection itself. This allows you to compute percentiles, group by namespace, correlate operation types with latency, and identify temporal patterns.

// Compute 95th percentile latency per namespace in the last hour
db.system.profile.aggregate([
  { $match: { 
      ts: { $gte: new Date(Date.now() - 3600000) },
      op: { $in: ["query", "command"] } 
  }},
  { $group: {
      _id: "$ns",
      avg_millis: { $avg: "$millis" },
      max_millis: { $max: "$millis" },
      p95_millis: { 
        $percentile: { 
          input: "$millis", 
          method: "approximate", 
          p: [0.95] 
        } 
      },
      count: { $sum: 1 }
  }},
  { $sort: { p95_millis: -1 } },
  { $limit: 10 }
]);

// Identify operations with high docsExamined to nreturned ratios
db.system.profile.aggregate([
  { $match: { op: "query", nreturned: { $gt: 0 } } },
  { $project: {
      ns: 1,
      millis: 1,
      ratio: { $divide: ["$docsExamined", "$nreturned"] },
      command: 1,
      ts: 1
  }},
  { $match: { ratio: { $gt: 100 } } },
  { $sort: { ratio: -1 } },
  { $limit: 20 }
]);

Capturing Profiler Data for Offline Analysis

For long-term trend analysis or cross-instance comparisons, export profiler data to external systems. You can use mongodump targeting the system.profile collection, or stream data continuously using change streams or scheduled aggregation queries that write to a separate analytics database.

// Export profiler data to a JSON file for offline analysis
// Run from the command line:
// mongodump --db ecommerce_db --collection system.profile \
//   --query '{"ts":{"$gte":{"$date":"2025-01-01T00:00:00Z"}}}' \
//   --out ./profiler_export/

// Alternatively, periodically archive slow queries to another collection
db.system_profile_archive.insertMany(
  db.system.profile.find({
    millis: { $gt: 200 },
    ts: { $gte: archiveStartDate, $lt: archiveEndDate }
  }).toArray()
);

Conclusion

MongoDB application performance profiling and optimization is an iterative, data-driven discipline. The profiler provides the raw evidence — revealing slow queries, collection scans, and inefficient patterns. The explain() method validates hypotheses about query plans and index usage. Together, they empower developers to move from reactive firefighting to proactive performance engineering. By following the structured workflow outlined in this tutorial — enabling profiling at an appropriate level, analyzing the system.profile collection systematically, using explain() to validate optimizations, designing indexes guided by the ESR rule, and continuously monitoring performance metrics — you can maintain responsive, scalable MongoDB applications even as data volumes and query complexity grow. Remember that optimization is not a one-time task but an ongoing practice: data evolves, access patterns shift, and what performs well today may need tuning tomorrow. Cultivate the habit of regular profiling review, index auditing, and query plan validation as core parts of your development lifecycle.