← Back to DevBytes
Testing July 18, 2026 15 min read

Mutation Testing: MutPy, Stryker: Complete Testing Guide for Developers

Mutation Testing: Beyond Code Coverage

Mutation testing is a powerful technique that evaluates the quality of your test suite by intentionally introducing bugs (mutations) into your source code and checking whether your tests detect them. Unlike traditional code coverage, which only tells you which lines were executed, mutation testing answers the critical question: How effective are your tests at catching real bugs?

What is Mutation Testing?

Mutation testing works by creating slightly modified versions of your source code, called mutants. Each mutant contains a single, small change—such as flipping a boolean operator, changing a comparison from > to >=, or removing a function call. These mutants are then run against your test suite. If a test fails, the mutant is killed (the test detected the bug). If all tests pass, the mutant survived (your tests missed the bug).

The ratio of killed mutants to total mutants is called the mutation score, and it serves as a far more meaningful metric than line or branch coverage alone. A high mutation score indicates that your test suite is robust and genuinely capable of catching regressions.

Why Does Mutation Testing Matter?

Traditional code coverage metrics can be misleading. Consider a test suite with 95% line coverage—it looks excellent on paper. However, that 95% coverage could be achieved by tests that simply execute code paths without making any meaningful assertions. Mutation testing exposes this gap:

The Core Concepts: Mutants, Killed vs Survived

When you run mutation testing, every mutant falls into one of these categories:

The mutation score is calculated as: (killed mutants) / (total mutants - equivalent mutants). Aiming for a score above 80% is generally considered strong, though for safety-critical systems, teams often target 90% or higher.

MutPy: Mutation Testing for Python

MutPy is a mutation testing framework for Python that modifies your source code at the AST (Abstract Syntax Tree) level and runs your test suite against each mutant. It supports unittest and pytest test discovery out of the box.

Installing MutPy

Install MutPy via pip. It requires Python 3.6 or later:

pip install mutpy

To verify the installation and see available options:

mutpy --help

Running MutPy on Your Project

The basic syntax runs mutation testing on a Python module and its corresponding test file:

mutpy --target calculator.py --unit-test test_calculator.py

For larger projects, you can specify entire directories. MutPy will discover test files automatically if you use naming conventions like test_*.py:

mutpy --target src/ --unit-test tests/ --runner pytest

Key Configuration Options

MutPy offers extensive configuration to tailor the mutation testing process. Here are the most important flags:

Complete MutPy Example

Let's walk through a complete example. First, here's a simple calculator module that we want to test:

# calculator.py

def add(a, b):
    """Return the sum of two numbers."""
    return a + b

def divide(a, b):
    """Return a divided by b. Raises ValueError if b is zero."""
    if b == 0:
        raise ValueError("Cannot divide by zero")
    return a / b

def is_positive(number):
    """Return True if number is positive, False otherwise."""
    return number > 0

def factorial(n):
    """Return the factorial of n. Raises ValueError if n < 0."""
    if n < 0:
        raise ValueError("Factorial is not defined for negative numbers")
    if n == 0:
        return 1
    result = 1
    for i in range(1, n + 1):
        result *= i
    return result

Now here's the test file. Notice that some tests have weak assertions—this will become apparent when we run mutation testing:

# test_calculator.py

import pytest
from calculator import add, divide, is_positive, factorial


def test_add_basic():
    assert add(2, 3) == 5
    assert add(-1, 1) == 0


def test_divide_basic():
    assert divide(10, 2) == 5


def test_divide_by_zero():
    with pytest.raises(ValueError):
        divide(10, 0)


def test_is_positive_true():
    assert is_positive(5) is True


def test_is_positive_false():
    assert is_positive(-3) is False
    # Missing test for zero! Zero is not positive, but do we test it?


def test_factorial_basic():
    assert factorial(5) == 120
    assert factorial(0) == 1


def test_factorial_negative():
    with pytest.raises(ValueError):
        factorial(-5)

Run MutPy against this code:

mutpy --target calculator.py --unit-test test_calculator.py --runner pytest --show-mutants

MutPy will produce output similar to the following (abbreviated for clarity):

MutPy Mutation Testing Report

Target: calculator.py
Test: test_calculator.py
Runner: pytest

[MUTATION #1] AOR - Arithmetic Operator Replacement
  Original:     return a + b
  Mutant:       return a - b
  Status: KILLED (test_add_basic failed)

[MUTATION #2] ROR - Relational Operator Replacement
  Original:     if b == 0:
  Mutant:       if b != 0:
  Status: KILLED (test_divide_by_zero failed)

[MUTATION #3] ROR - Relational Operator Replacement
  Original:     return number > 0
  Mutant:       return number >= 0
  Status: SURVIVED
  Note: No test distinguishes between > and >= for zero input.
        Add a test for is_positive(0).

[MUTATION #4] ROR - Relational Operator Replacement
  Original:     if n < 0:
  Mutant:       if n <= 0:
  Status: KILLED (test_factorial_negative failed)

[MUTATION #5] ROR - Relational Operator Replacement
  Original:     if n == 0:
  Mutant:       if n != 0:
  Status: SURVIVED
  Note: The mutant causes factorial(0) to skip the base case,
        but no test explicitly catches this. Consider adding
        an assertion for factorial(0) == 1.

--- Summary ---
Total mutants: 12
Killed: 9
Survived: 2
Equivalent: 1
Mutation score: 81.8%

The report reveals critical gaps. Mutation #3 survived because we never tested is_positive(0). The mutation changed > to >=, which would make is_positive(0) incorrectly return True, yet no test caught it. Mutation #5 survived because although we tested factorial(0), the specific assertion wasn't strong enough to detect the boundary change.

MutPy Mutation Operators Reference

MutPy supports numerous mutation operators. Here are the most commonly used ones:

To apply only specific operators, use the --only-mutate flag:

mutpy --target calculator.py --unit-test test_calculator.py --only-mutate ROR LOR

Generating HTML Reports with MutPy

For team review or CI integration, generate a detailed HTML report:

mutpy --target src/ --unit-test tests/ --runner pytest --report html --report-path mutation_report.html

The HTML report includes color-coded source files showing exactly which lines contain surviving mutants, making it easy to prioritize test improvements.

MutPy Best Practices Summary

Stryker: Mutation Testing for JavaScript and TypeScript

Stryker is a modern, highly configurable mutation testing framework for JavaScript, TypeScript, and related ecosystems. It supports all major test runners including Jest, Mocha, Jasmine, Karma, and Vitest, and works with both Node.js and browser-based projects.

Installing Stryker

Stryker is installed as a dev dependency via npm or yarn. The recommended approach uses the Stryker initialization wizard, which sets up the configuration file automatically:

# Install Stryker core and the appropriate runner plugin
npm install --save-dev @stryker-mutator/core

# Run the setup wizard (recommended for first-time users)
npx stryker init

The wizard will ask you a series of questions about your project—which test runner you use, which package manager, whether you use TypeScript, and where your source and test files are located. It then generates a stryker.config.json (or .js) file.

Alternatively, you can install specific plugins directly. Here's a typical setup for a Jest + TypeScript project:

npm install --save-dev @stryker-mutator/core @stryker-mutator/jest-runner @stryker-mutator/typescript-plugin

Creating the Stryker Configuration

The configuration file is the heart of Stryker. Here's a complete example for a Node.js project using Jest:

// stryker.config.json
{
  "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
  "mutator": {
    "plugins": ["javascript-mutator"],
    "excludedMutations": ["StringLiteral"]
  },
  "testRunner": "jest",
  "jest": {
    "projectType": "custom",
    "configFile": "jest.config.js"
  },
  "reporters": ["html", "clear-text", "progress"],
  "htmlReporter": {
    "fileName": "mutation-report.html"
  },
  "thresholds": {
    "high": 80,
    "low": 60,
    "break": null
  },
  "tempDirName": ".stryker-tmp",
  "cleanTempDir": true,
  "timeoutMS": 30000,
  "concurrency": 4
}

Key configuration sections explained:

Complete Stryker Example

Let's walk through a realistic Node.js example. Here's a user service module that handles authentication logic:

// userService.js

function validatePassword(password) {
  // Password must be at least 8 characters and contain a number
  if (password.length >= 8 && /\d/.test(password)) {
    return { valid: true };
  }
  return { valid: false, reason: 'Password must be at least 8 characters and contain a number' };
}

function calculateDiscount(user) {
  if (user.isPremium && user.purchaseCount > 10) {
    return 0.15; // 15% discount
  }
  if (user.isPremium || user.purchaseCount > 5) {
    return 0.05; // 5% discount
  }
  return 0;
}

module.exports = { validatePassword, calculateDiscount };

Here's the corresponding test file. It has decent coverage but contains subtle gaps:

// userService.test.js

const { validatePassword, calculateDiscount } = require('./userService');

describe('validatePassword', () => {
  test('rejects short password', () => {
    const result = validatePassword('abc1');
    expect(result.valid).toBe(false);
  });

  test('rejects password without number', () => {
    const result = validatePassword('abcdefghij');
    expect(result.valid).toBe(false);
  });

  test('accepts valid password', () => {
    const result = validatePassword('password1');
    expect(result.valid).toBe(true);
  });
});

describe('calculateDiscount', () => {
  test('gives 15% discount to premium users with many purchases', () => {
    const user = { isPremium: true, purchaseCount: 15 };
    expect(calculateDiscount(user)).toBe(0.15);
  });

  test('gives 5% discount to premium users with few purchases', () => {
    const user = { isPremium: true, purchaseCount: 3 };
    expect(calculateDiscount(user)).toBe(0.05);
  });

  test('gives 5% discount to regular users with many purchases', () => {
    const user = { isPremium: false, purchaseCount: 8 };
    expect(calculateDiscount(user)).toBe(0.05);
  });

  test('gives no discount to regular users with few purchases', () => {
    const user = { isPremium: false, purchaseCount: 2 };
    expect(calculateDiscount(user)).toBe(0);
  });
});

Run Stryker:

npx stryker run

The console output will look something like this:

----------- Mutation testing report -----------

All tests passed.

Mutants killed: 18
Mutants survived: 3
Mutants timed out: 0
Mutants with no coverage: 0
Total mutants: 21
Mutation score (based on covered code): 85.71%

Surviving mutants:

#1. userService.js:3:27
  Mutator: EqualityOperator
  - if (password.length >= 8 && /\d/.test(password)) {
  + if (password.length > 8 && /\d/.test(password)) {
  Tests passed: all
  Hint: Add a test for a password exactly 8 characters long.

#2. userService.js:12:15
  Mutator: LogicalOperator
  - if (user.isPremium || user.purchaseCount > 5) {
  + if (user.isPremium && user.purchaseCount > 5) {
  Tests passed: all
  Hint: The boundary between the two discount tiers is not tested
         thoroughly. Consider edge cases where both conditions overlap.

#3. userService.js:14:9
  Mutator: BlockStatement
  - return 0;
  + (empty block)
  Tests passed: all
  Hint: The fallback return 0 is not validated independently;
         ensure the default case is explicitly tested.

The surviving mutants reveal precise gaps. The first survivor shows that we never tested a password exactly 8 characters long (the boundary). The second reveals that our tests don't distinguish between the || and && logic at the threshold boundary—a user who is both premium and has exactly 5 purchases might get the wrong discount. The third survivor indicates that the default return value of 0 is never independently verified.

Stryker Mutators Reference

Stryker includes dozens of mutators organized by category. Here are the most important ones:

You can exclude specific mutators in the configuration:

{
  "mutator": {
    "excludedMutations": ["StringLiteral", "ArrayDeclaration"]
  }
}

Stryker with TypeScript

For TypeScript projects, Stryker requires the TypeScript plugin and uses the typescript-mutator. Your configuration should look like this:

// stryker.config.json for TypeScript + Jest
{
  "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
  "mutator": {
    "plugins": ["typescript-mutator"],
    "excludedMutations": ["StringLiteral"]
  },
  "testRunner": "jest",
  "jest": {
    "projectType": "custom",
    "configFile": "jest.config.js"
  },
  "reporters": ["html", "clear-text", "progress"],
  "tsconfigFile": "tsconfig.json",
  "thresholds": {
    "high": 80,
    "low": 60,
    "break": 70
  },
  "timeoutMS": 30000,
  "concurrency": 4
}

Run it the same way:

npx stryker run

Stryker Incremental Mode

For large codebases, running full mutation testing on every commit can be slow. Stryker supports incremental mutation testing, which only mutates files that have changed since the last run:

npx stryker run --incremental

Stryker stores the previous run results in a .stryker-tmp/incremental directory and uses git diff information to determine which files need re-mutation. This can reduce execution time by 70-90% in CI pipelines.

Stryker Dashboard Integration

Stryker offers a free cloud dashboard service that tracks your mutation score over time. Enable it by adding the dashboard reporter:

{
  "reporters": ["dashboard", "html", "clear-text"]
}

Then set the required environment variables in your CI environment:

# Environment variables for Stryker Dashboard
export STRYKER_DASHBOARD_API_KEY="your-project-api-key"
export STRYKER_DASHBOARD_PROJECT="github.com/your-org/your-repo"
export STRYKER_DASHBOARD_VERSION="1.0.0"

This allows you to view historical mutation scores, compare branches, and set regression thresholds directly in the dashboard.

Best Practices for Mutation Testing

1. Start Small, Then Scale

Mutation testing is computationally expensive. For a large codebase, a full run can take hours. Begin by running mutation testing on your most critical modules—authentication, payment processing, data validation—and gradually expand coverage as you optimize your test suite and CI pipeline.

2. Set Realistic Thresholds

A 100% mutation score is rarely achievable due to equivalent mutants. Aim for 80-85% as a strong baseline, and increase thresholds incrementally as your team gains experience. Use Stryker's thresholds.break or MutPy's exit codes to enforce quality gates in CI.

3. Treat Surviving Mutants as Actionable Bugs

Every surviving mutant represents a potential undetected bug. Create a workflow where the team reviews the HTML mutation report, triages survivors, and writes the missing tests before the next sprint. This turns mutation testing from a metrics tool into a concrete quality improvement process.

4. Exclude Equivalent Mutants Judiciously

Equivalent mutants—mutations that produce identical behavior—are inevitable. For example, mutating i++ to i += 1 or i = i + 1 produces an equivalent mutant. Both MutPy and Stryker allow you to exclude specific mutations or entire mutation operators. Document why each exclusion exists to prevent accidentally suppressing important mutation types.

5. Combine Mutation Testing with Code Review

The HTML reports generated by both tools are excellent artifacts for code review. Before merging a pull request, review the mutation report to see if new code introduces survivable mutants. This catches testing gaps before they enter the main branch.

6. Run Mutation Testing in CI, Not Locally (for Large Projects)

For projects with thousands of tests, mutation testing is best suited for CI pipelines where parallelization and dedicated compute resources can handle the load. Configure your CI to run mutation testing on merge to main or as a nightly job, and report results back to the team via the dashboard or report artifacts.

7. Use Incremental Mode in CI

Stryker's incremental mode and MutPy's targeted runs (using --target on changed files) dramatically reduce execution time. Integrate these with your CI's change detection to run mutation testing only on modified code paths.

8. Educate the Team on Mutation Types

Understanding why a mutant survived is more valuable than the score itself. Train developers to interpret mutation reports: a surviving LogicalOperator mutant points to missing edge case tests, while a surviving BlockStatement mutant suggests dead code or insufficient integration testing. This knowledge directly improves testing skills across the team.

Conclusion

Mutation testing with MutPy and Stryker transforms how you think about test quality. It moves the conversation beyond the misleading comfort of high code coverage percentages and into the realm of genuine test effectiveness. By introducing controlled faults and measuring your test suite's ability to detect them, you gain a precise, actionable map of your testing weaknesses. Both tools integrate smoothly into modern development workflows—MutPy for Python projects with pytest or unittest, and Stryker for JavaScript and TypeScript projects with any major test runner. Start with a single critical module, set a reasonable mutation score threshold in CI, and build the habit of reviewing surviving mutants with every release. The result is a test suite you can truly trust, and a codebase that is measurably more resilient to regressions.

🚀 Need a reliable AI agent for your project?

Deploy Hermes Agent in 10 minutes. Managed hosting, zero DevOps.

Get Started — $23.99/mo
← Back to all articles