Emacs Testing Integration: Complete Guide

What Is Emacs Testing Integration

Emacs Testing Integration refers to the built-in and third-party frameworks that allow developers to write, run, and manage automated tests directly within the Emacs editor. At its core sits ERT (Emacs Lisp Regression Testing), a testing framework bundled with Emacs since version 24.1. ERT provides a structured way to define test cases, make assertions, and verify that your Elisp code behaves correctly across revisions.

Beyond ERT, the Emacs ecosystem offers several complementary testing layers:

• Buttercup – a behavior-driven testing framework inspired by Ruby's RSpec, offering a more expressive syntax
• ert-runner – a CLI tool that runs ERT tests outside the Emacs GUI, ideal for CI/CD pipelines
• Eldev – a modern build tool and package manager with integrated test-running capabilities
• Makem – a lightweight Makefile-based test runner for batch execution

Together, these tools form a comprehensive testing ecosystem that transforms Emacs from a mere text editor into a full-fledged Elisp development environment with continuous feedback loops.

Why Testing Integration Matters

For Emacs Lisp developers—whether you're writing a small helper function or a complex package distributed on MELPA—automated testing is not optional. Here's why:

• Regression prevention – Emacs packages often interact with global state, hooks, and buffers. A seemingly innocent change can break functionality weeks later. Tests catch this immediately.
• Refactoring confidence – Elisp's dynamic nature makes it easy to introduce subtle bugs during refactoring. A solid test suite lets you restructure code fearlessly.
• Package acceptance – MELPA and ELPA maintainers increasingly expect packages to include tests. A well-tested package signals quality and reliability.
• Interactive development feedback – Running tests inside Emacs gives you instant feedback without leaving your editing context. You can fix a function, hit C-c C-t, and see results immediately.
• CI/CD compatibility – With tools like ert-runner and Eldev, you can run the same test suite on GitHub Actions, GitLab CI, or any build server, ensuring every commit is validated.

Getting Started with ERT

Writing Your First ERT Test

ERT tests live in files typically named with a -tests.el suffix. Here's a minimal example that tests a simple string utility function:

;; my-string-utils.el -- A simple string utility library
;;; Code:

(defun capitalize-words (str)
  "Capitalize the first letter of each word in STR."
  (let ((words (split-string str " ")))
    (mapconcat
     (lambda (word)
       (concat (capitalize (substring word 0 1))
               (substring word 1)))
     words
     " ")))

(provide 'my-string-utils)
;;; my-string-utils.el ends here

Now create the corresponding test file:

;; my-string-utils-tests.el -- Tests for my-string-utils
;;; Code:

(require 'ert)
(require 'my-string-utils)

(ert-deftest capitalize-words-basic ()
  "Test basic word capitalization."
  (should (equal (capitalize-words "hello world")
                 "Hello World")))

(ert-deftest capitalize-words-single-word ()
  "Test capitalization of a single word."
  (should (equal (capitalize-words "emacs")
                 "Emacs")))

(ert-deftest capitalize-words-empty-string ()
  "Test that an empty string returns an empty string."
  (should (equal (capitalize-words "")
                 "")))

(ert-deftest capitalize-words-mixed-case ()
  "Test words that are already mixed case."
  (should (equal (capitalize-words "the QUICK brown")
                 "The QUICK Brown")))

(provide 'my-string-utils-tests)
;;; my-string-utils-tests.el ends here

Running ERT Tests Inside Emacs

There are multiple ways to execute ERT tests:

• Interactive – Evaluate the test file with M-x eval-buffer, then run M-x ert-run-tests-interactively and select the tests you want. You'll see a buffer with colored pass/fail results.
• Batch mode – From the command line: emacs -batch -l my-string-utils-tests.el -f ert-run-tests-batch-and-exit
• Via keybinding – With ert-run-tests-interactively, you can filter by test name using the prompt.

Advanced ERT Assertions

ERT provides several assertion macros beyond should:

(ert-deftest advanced-assertions ()
  "Demonstrate various ERT assertion types."

  ;; Basic equality
  (should (= (+ 2 2) 4))

  ;; Type checking
  (should (stringp "hello"))
  (should (not (numberp "hello")))

  ;; Error testing -- verifies that an error is signaled
  (should-error (car "not-a-list") :type 'wrong-type-argument)

  ;; Testing specific error conditions
  (should-error
   (let ((x "string")) (+ x 1))
   :type 'wrong-type-argument)

  ;; should-not is the negation of should
  (should-not (equal nil t))

  ;; should-equal for detailed diff output on failure
  (should-equal
   (list 1 2 3 4)
   (mapcar 'identity '(1 2 3 4))))

Fixtures and Setup/Teardown

For tests requiring specific environment setup, use ert-deftest with let-bindings or dedicated fixture macros:

(ert-deftest with-temp-buffer-test ()
  "Test buffer-local operations using a temporary buffer."
  (with-temp-buffer
    (insert "Line 1\nLine 2\nLine 3\n")
    (should (= (count-lines (point-min) (point-max)) 3))
    (should (equal (buffer-string)
                   "Line 1\nLine 2\nLine 3\n"))))

;; Using a custom fixture macro for repeated setup
(defmacro my-test-with-clean-environment (&rest body)
  "Execute BODY with a clean, isolated environment."
  `(let ((inhibit-message t)
         (gc-cons-threshold most-positive-fixnum))
     (with-temp-buffer
       ,@body)))

(ert-deftest isolated-computation-test ()
  "Test computation in an isolated environment."
  (my-test-with-clean-environment
   (let ((result (my-expensive-computation)))
     (should result)
     (should (>= result 0)))))

Working with Buttercup

Installing Buttercup

Buttercup is available on MELPA. Install it via M-x package-install RET buttercup RET, or add it to your Cask file for project-level dependency management:

;; Cask file for your project
(source gnu)
(source melpa)

(package-file "my-package.el")

(development
 (depends-on "buttercup"))

Writing Buttercup Specs

Buttercup uses a behavior-driven syntax with describe blocks and it specifications. This style encourages thinking about expected behavior rather than implementation details:

;; my-package-tests.el -- Buttercup specs for my-package
;;; Code:

(require 'buttercup)
(require 'my-package)

(describe "my-package"
  (describe "parse-config-file"
    (it "should return nil for non-existent files"
      (expect (my-package-parse-config-file "/nonexistent/path")
              :to-be nil))

    (it "should parse a valid JSON config"
      (with-temp-buffer
        (insert "{\"name\": \"test\", \"version\": 1}")
        (write-region (point-min) (point-max) "/tmp/test-config.json")
        (expect (my-package-parse-config-file "/tmp/test-config.json")
                :to-equal '((name . "test") (version . 1)))))

    (it "should signal an error on malformed input"
      (expect (my-package-parse-config-file "/dev/null")
              :to-throw 'my-package-parse-error)))

  (describe "format-output"
    (it "should format a simple list as a table"
      (expect (my-package-format-output '((a . 1) (b . 2)))
              :to-match "| a | 1 |"))

    (it "should handle empty input gracefully"
      (expect (my-package-format-output nil)
              :to-be ""))))

(provide 'my-package-tests)
;;; my-package-tests.el ends here

Running Buttercup Tests

Execute Buttercup specs inside Emacs with M-x buttercup-run or from the command line:

# Run all specs in a directory
emacs -batch -l buttercup -l my-package-tests.el -f buttercup-run

# Run with buttercup CLI wrapper (if installed)
buttercup -L . --pattern "parse-config-file"

Continuous Integration with ert-runner

ert-runner is a dedicated test runner for ERT that operates outside the Emacs GUI. It's ideal for CI servers. Install it via MELPA or as a standalone script:

# Install ert-runner globally
curl -fsSL https://raw.githubusercontent.com/ecukes/ert-runner/master/bin/ert-runner -o /usr/local/bin/ert-runner
chmod +x /usr/local/bin/ert-runner

Project Structure for ert-runner

ert-runner expects a specific directory layout:

my-project/
├── my-project.el          # Main package file
├── Cask                   # Dependency manifest
├── Makefile               # Build automation
└── test/
    ├── my-project-test.el # Primary test file
    └── helpers.el         # Test utilities and fixtures

Sample Makefile Integration

# Makefile for Emacs package testing
EMACS ?= emacs
CASK ?= cask
ERT_RUNNER ?= ert-runner

.PHONY: test unit-test integration-test clean

# Run all tests
test: unit-test

unit-test:
	$(CASK) exec $(ERT_RUNNER) --reporter ert-runner-reporter-dot

integration-test:
	$(CASK) exec $(ERT_RUNNER) --pattern integration --reporter ert-runner-reporter-dot

# Run with verbose output for debugging
test-verbose:
	$(CASK) exec $(ERT_RUNNER) --verbose --no-color

clean:
	rm -rf .cask/
	$(CASK) pristine-clean

GitHub Actions Workflow

# .github/workflows/test.yml
name: Emacs Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    strategy:
      matrix:
        emacs-version: [26.3, 27.2, 28.2, 29.1]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Install Emacs
        uses: purcell/setup-emacs@master
        with:
          version: ${{ matrix.emacs-version }}

      - name: Install Cask
        run: |
          curl -fsSL https://raw.githubusercontent.com/cask/cask/master/go | python
          export PATH="$HOME/.cask/bin:$PATH"

      - name: Install dependencies
        run: cask install

      - name: Run tests
        run: make test

Using Eldev for Modern Test Management

Eldev is a more modern alternative to Cask + ert-runner. It bundles dependency management, building, and testing into a single tool. Install it globally:

# Install Eldev
curl -fsSL https://raw.githubusercontent.com/emacs-eldev/eldev/master/webinstall/eldev-install -o /tmp/eldev-install
bash /tmp/eldev-install

Running Tests with Eldev

# Run all ERT tests
eldev test

# Run tests matching a pattern
eldev test --pattern "my-package-"

# Run with Buttercup support
eldev test --runner buttercup

# Run in batch mode for CI
eldev test --mode batch

# Generate a test coverage report
eldev test --coverage

Eldev Project File

Eldev uses an Eldev file instead of Cask:

;; Eldev file for project configuration
(eldev-use-package-archive 'gnu)
(eldev-use-package-archive 'melpa)

(eldev-package-file "my-package.el")

(eldev-add-extra-dependency 'buttercup "1.24")
(eldev-add-extra-dependency 'f "0.20.0")

Testing Interactive Commands

Testing interactive Emacs commands presents unique challenges because they often manipulate buffers, read user input, or modify global state. ERT provides utilities to simulate these interactions:

(ert-deftest test-interactive-command ()
  "Test an interactive command that operates on a region."
  (with-temp-buffer
    (insert "Hello World\nGoodbye Moon\n")
    ;; Simulate marking a region
    (goto-char 5)
    (set-mark (point))
    (goto-char 12)
    ;; Activate the region and call the command
    (let ((mark-active t))
      (call-interactively 'my-uppercase-region))
    ;; Verify the transformation
    (should (equal (buffer-string)
                   "HELLO World\nGoodbye Moon\n"))))

;; Testing commands that read from the minibuffer
(ert-deftest test-minibuffer-interaction ()
  "Test a command that prompts for user input."
  (cl-letf (((symbol-function 'read-string)
             (lambda (_prompt &optional _initial _history _default _inherit)
               "test-input")))
    (let ((result (my-prompt-and-process)))
      (should (equal result "PROCESSED: test-input")))))

;; Testing with simulated key sequences
(ert-deftest test-keybinding-command ()
  "Test a command normally invoked via keybinding."
  (with-temp-buffer
    (insert "sample text")
    ;; Simulate key press that invokes the command
    (execute-kbd-macro (kbd "M-x my-transform-text RET"))
    (should (equal (buffer-string) "TRANSFORMED: sample text"))))

Mocking and Stubbing in Elisp Tests

Elisp's dynamic scoping makes mocking remarkably straightforward. You can temporarily rebind functions using cl-letf:

(ert-deftest test-with-network-mock ()
  "Test a function that normally makes HTTP requests."
  ;; Mock url-retrieve-synchronously to return fake data
  (cl-letf (((symbol-function 'url-retrieve-synchronously)
             (lambda (_url &optional _silent _inhibit-cookies)
               (with-temp-buffer
                 (insert "HTTP/1.1 200 OK\n\n{\"status\": \"ok\"}")
                 (current-buffer)))))
    (let ((result (my-package-fetch-status "https://api.example.com")))
      (should (equal result '((status . "ok")))))))

;; Mocking a process-related function
(ert-deftest test-process-communication ()
  "Test function that communicates with an external process."
  (cl-letf (((symbol-function 'call-process)
             (lambda (_program &optional _infile _display _wait &rest _args)
               (with-temp-buffer
                 (insert "output line 1\noutput line 2\n")
                 (current-buffer)))))
    (let ((lines (my-package-run-external-tool "some-tool")))
      (should (equal lines '("output line 1" "output line 2")))))))

Test Organization Best Practices

File Structure

A well-organized test directory scales with your package:

my-package/
├── my-package.el
├── my-package-utils.el
├── my-package-ui.el
├── Cask / Eldev
├── Makefile
└── test/
    ├── helpers.el              # Shared test utilities
    ├── my-package-unit-tests.el
    ├── my-package-integration-tests.el
    ├── my-package-ui-tests.el
    └── fixtures/
        ├── sample-config.json
        └── mock-data.el

Naming Conventions

Use descriptive test names that reveal intent at a glance:

;; GOOD: Clear, descriptive names
(ert-deftest parse-json-returns-nil-on-empty-input ()
  ...)

(ert-deftest parse-json-signals-error-on-malformed-input ()
  ...)

(ert-deftest format-date-handles-leap-year-correctly ()
  ...)

;; AVOID: Vague or implementation-coupled names
(ert-deftest test-1 ()
  ...)

(ert-deftest parse-json-test-2 ()
  ...)

Tagging Tests for Selective Execution

ERT supports tagging tests to run subsets selectively:

;; Tag tests with metadata for selective execution
(ert-deftest slow-network-test ()
  :tags '(slow network integration)
  ;; This test makes actual network calls
  ...)

(ert-deftest fast-unit-test ()
  :tags '(fast unit)
  ;; Pure computation, no I/O
  ...)

;; Run only fast tests
;; M-x ert-run-tests-interactively, then filter by tag "fast"
;; Or from batch: emacs -batch -l my-tests.el -f ert-run-tests-batch-and-exit fast

Helper Utilities

Extract common patterns into a shared helpers file:

;; test/helpers.el -- Shared test utilities
;;; Code:

(require 'ert)

(defmacro with-test-sandbox (&rest body)
  "Execute BODY in a clean temporary directory.
Prevents tests from polluting the user's filesystem."
  `(let ((default-directory (make-temp-file "emacs-test-" 'dir)))
     (unwind-protect
         (progn ,@body)
       (delete-directory default-directory 'recursive))))

(defun create-test-file (content &optional filename)
  "Create a temporary file with CONTENT for testing purposes."
  (let ((file (or filename (make-temp-file "emacs-test-"))))
    (with-temp-buffer
      (insert content)
      (write-region (point-min) (point-max) file))
    file))

(defmacro with-mocked-function (func-name replacement &rest body)
  "Temporarily replace FUNC-NAME with REPLACEMENT during BODY."
  `(cl-letf (((symbol-function ,func-name) ,replacement))
     ,@body))

(provide 'helpers)
;;; helpers.el ends here

Debugging Failing Tests

When a test fails, Emacs provides several powerful debugging facilities:

Using the ERT Failure Buffer

After running tests with ert-run-tests-interactively, click on a failing test in the *ert* buffer. Emacs will show the expected vs. actual values. For complex failures, use ert-results-find-test-at-point to jump directly to the test definition.

Interactive Debugging with Edebug

;; Instrument a test for debugging
(ert-deftest debug-this-test ()
  "A test you want to step through with Edebug."
  ;; Place point inside the test and press C-u C-M-x to instrument it
  (let ((complex-data (my-function-that-fails)))
    (should (equal complex-data '(:expected :structure)))))

With the test instrumented, run it via ert-run-tests-interactively and Emacs will drop into Edebug at the first instrumented form, allowing you to step through execution, inspect variables, and identify the exact point of failure.

Adding Diagnostic Messages

(ert-deftest test-with-diagnostics ()
  "Use messages to trace test execution."
  (let ((input '((a . 1) (b . 2))))
    (message "DEBUG: Input is %S" input)
    (let ((result (my-transform input)))
      (message "DEBUG: Result is %S" result)
      (should (equal result '((a . 2) (b . 4)))))))

Performance Testing and Benchmarks

Emacs includes a built-in benchmarking facility that integrates with ERT:

(ert-deftest performance-regression-check ()
  "Ensure critical operations remain fast."
  (let ((large-list (make-list 10000 '(:key . "value"))))
    (let ((elapsed (benchmark-run 10
                     (my-parsing-function large-list))))
      ;; After 10 runs, average time should be under 0.01 seconds
      (should (< (/ (car elapsed) 10) 0.01)))))

;; Dedicated benchmark test
(ert-deftest benchmark-sorting-algorithm ()
  "Benchmark and validate sorting performance."
  (let* ((data (cl-loop for i from 1 to 5000
                        collect (cons i (random 1000))))
         (timing (benchmark-run 1
                   (my-custom-sort data))))
    (message "Sorting 5000 items took %s seconds" (car timing))
    ;; Assert it completes within reasonable time
    (should (< (car timing) 2.0))))

Code Coverage Analysis

While Emacs doesn't have built-in code coverage, you can integrate with external tools or use simple manual instrumentation:

;; Simple coverage tracking for critical code paths
(defvar my-coverage-tracker (make-hash-table :test 'equal)
  "Hash table tracking which functions are called during tests.")

(defmacro track-coverage (funcall-form)
  "Execute FUNCALL-FORM and record coverage."
  `(progn
     (puthash ',funcall-form t my-coverage-tracker)
     ,funcall-form))

(ert-deftest verify-coverage ()
  "Check that all critical code paths were exercised."
  ;; Run your test suite first, then:
  (should (gethash '(my-critical-function arg1 arg2) my-coverage-tracker))
  (should (gethash '(my-error-handler invalid-input) my-coverage-tracker))
  (should (gethash '(my-edge-case-handler nil) my-coverage-tracker)))

Best Practices Summary

• Write tests alongside code, not after – Test-driven development in Elisp is productive because the feedback loop is instantaneous inside Emacs.
• Keep tests isolated – Each test should set up its own state and clean up afterward. Avoid relying on test execution order.
• Test edge cases first – Empty inputs, nil values, boundary conditions, and error paths often hide the most insidious bugs.
• Use descriptive test names – A test name should describe the expected behavior so clearly that it serves as documentation.
• Tag tests for granular execution – Use :tags to separate fast unit tests from slow integration tests, enabling quick feedback during development.
• Run tests in batch mode for CI – Always verify that your test suite passes in batch mode (emacs -batch) before pushing, as interactive-only tests may fail on CI servers.
• Mock external dependencies – Network calls, file system operations, and process invocations should be mocked in unit tests to keep them fast and deterministic.
• Version-control your test fixtures – Sample data files, mock JSON, and test configurations belong in version control alongside your tests.
• Integrate testing into your workflow – Bind ert-run-tests-interactively to a key, use save-some-buffers hooks to auto-run tests, or configure Flycheck to run tests on save.
• Document how to run your tests – Include a Makefile target or a README section explaining exactly how to execute the test suite, so contributors can validate their changes easily.

Conclusion

Emacs Testing Integration transforms the editor into a robust Elisp development environment where quality assurance is not an afterthought but an integral part of the workflow. From ERT's straightforward assertion-based tests to Buttercup's expressive behavior-driven specs, from ert-runner's CI-friendly batch execution to Eldev's modern all-in-one approach, the ecosystem provides tools for every testing philosophy and project scale. By adopting these practices—writing isolated, well-named tests, mocking external dependencies, tagging for selective execution, and integrating testing into both your local workflow and CI pipeline—you build Emacs packages that are maintainable, reliable, and ready for widespread distribution. The investment in learning these testing patterns pays dividends in reduced debugging time and increased confidence with every change you make.