Git Filter-Branch and Filter-Repo

Understanding Git History Rewriting

Every Git repository carries a complete history — every commit, every file change, every author timestamp. Sometimes that history needs to be altered. Maybe a secret key was committed, a repository grew bloated from large binary files, or you want to extract a single project folder into its own repository. Rewriting history is a powerful but delicate operation that permanently changes commit SHAs and rewrites the entire commit graph.

What is git filter-branch?

git filter-branch is a built-in Git command designed to rewrite branches by applying custom filters to each commit’s content, author information, message, or even the tree structure. It was for many years the standard tool for tasks like removing sensitive files from the entire history, fixing committer email addresses across thousands of commits, or extracting a subdirectory into a new repository root. However, it has significant performance problems and edge cases that can corrupt history, and Git maintainers now actively discourage its use.

What is git filter-repo?

git filter-repo is a third-party tool (officially endorsed by Git documentation) that replaces and vastly improves upon git filter-branch. Written in Python, it’s dramatically faster, safer, and easier to use. It eliminates many of the footguns present in filter-branch — no more confusing shell variable escaping, no more accidental ref updates that break clones, and a streamlined command-line interface focused on common rewriting tasks. It is now the recommended tool for all repository history rewriting.

Why Rewriting History Matters

History rewriting is not an everyday operation, but when it’s necessary, it’s critical. Common scenarios include:

• Removing sensitive data — API keys, credentials, proprietary code accidentally pushed to a public or shared repository.
• Reducing repository size — Large binary assets or archived files that were committed years ago and then removed, but still live in the packfile forever.
• Extracting a subfolder — A monorepo grows and a single project needs its own independent repository with full history.
• Fixing author/committer identity — Wrong email addresses or names that need to be corrected across the entire history before migrating to a new platform.
• Changing commit messages or paths — Bulk correcting a typo in a license file path or updating a URL embedded in commit messages.

Because rewriting changes every commit SHA, it effectively creates an entirely new repository history. This means it should only be performed on repositories where all collaborators have agreed to the rewrite, or better yet, on a fresh clone before the repository is shared widely.

Using git filter-branch

git filter-branch works by running a provided filter over every commit in the specified range. It supports several filter types:

• --env-filter — modify author/committer environment variables.
• --tree-filter — modify the working directory of each commit (very slow).
• --index-filter — modify the staging area without checking out files (much faster).
• --msg-filter — rewrite commit messages.
• --subdirectory-filter — extract a subdirectory as the new root.
• --tag-name-filter — transform tag names to point to rewritten commits.

Example: Removing a Sensitive File from History

Suppose you accidentally committed config/secrets.yml and need to purge it from every commit in the repository. Using git filter-branch:

git filter-branch --force --index-filter \
  'git rm --cached --ignore-unmatch config/secrets.yml' \
  --prune-empty --tag-name-filter cat -- --all

Explanation:

• --force — allows the command to run even if a previous filter-branch backup exists.
• --index-filter — runs the given command on the index (staging area) without checking out files, making it much faster than --tree-filter.
• git rm --cached --ignore-unmatch — removes the file from the index; --ignore-unmatch prevents errors on commits that never had the file.
• --prune-empty — discards commits that become empty after the file removal.
• --tag-name-filter cat — updates tags to point to the rewritten commits (the cat command just passes the tag name through unchanged).
• -- --all — rewrites all branches and tags.

After running, you will need to force-push the rewritten history if the repository is shared. Warning: This is a destructive operation for collaborators.

Example: Changing Author and Committer Information

To fix an old email address across all commits:

git filter-branch --env-filter '
  if [ "$GIT_COMMITTER_EMAIL" = "old@example.com" ]; then
    export GIT_COMMITTER_EMAIL="new@example.com"
  fi
  if [ "$GIT_AUTHOR_EMAIL" = "old@example.com" ]; then
    export GIT_AUTHOR_EMAIL="new@example.com"
  fi
' --tag-name-filter cat -- --branches --tags

The --env-filter runs a shell script where Git provides environment variables like GIT_COMMITTER_NAME, GIT_COMMITTER_EMAIL, GIT_AUTHOR_NAME, and GIT_AUTHOR_EMAIL. Changing them alters the commit metadata.

Example: Extracting a Subdirectory as a New Root

To turn the src/ folder into the root of a new repository:

git filter-branch --subdirectory-filter src/ -- --all

This rewrites history so that every commit’s tree starts at what was originally src/. Files outside that directory are completely discarded.

Limitations and Warnings with git filter-branch

Despite its power, git filter-branch has many known problems:

• Performance — --tree-filter checks out every commit, which is incredibly slow on large repositories. Even --index-filter can be sluggish.
• Shell escaping nightmares — Complex filters require careful quoting and escaping, often leading to subtle bugs.
• Backup ref pollution — It creates refs/original/ backup refs that must be manually cleaned up with git filter-branch -f or git update-ref -d.
• Deprecated — The official Git documentation now includes a warning that git filter-branch is deprecated and points users to git filter-repo.

Because of these issues, you should only use git filter-branch if you are maintaining legacy scripts and have no alternative. For all new work, use git filter-repo.

Using git filter-repo (The Modern Alternative)

git filter-repo is not shipped with Git; you must install it separately. The easiest way is via pip:

pip install git-filter-repo

Then you can run it as git filter-repo from inside any repository. It operates on the current clone and does not require a bare repository (though using a fresh clone is always recommended). The tool is designed to be used in a single invocation with clear, composable flags.

Example: Removing a File or Folder

To permanently delete a sensitive file, use --path and --invert-paths:

git filter-repo --path config/secrets.yml --invert-paths

This removes all traces of that file from every commit. To remove an entire directory:

git filter-repo --path old-binaries/ --invert-paths

You can specify multiple paths in one command. The --invert-paths flag tells filter-repo to keep everything except the listed paths.

Example: Changing Author Information

git filter-repo uses callback scripts (Python expressions) instead of fragile shell environment manipulations. To replace an old email:

git filter-repo --email-callback '
  if email == b"old@example.com":
      return b"new@example.com"
  return email
'

For changing both name and email, use --commit-callback:

git filter-repo --commit-callback '
  if commit.author_name == b"Old Name":
      commit.author_name = b"New Name"
      commit.author_email = b"new@example.com"
'

The b prefix denotes byte strings, which is how filter-repo internally handles commit data. This approach avoids any shell injection risks and is much more readable.

Example: Extracting a Subdirectory

To isolate a subfolder and make it the new root, combine --path and --subdirectory-filter:

git filter-repo --path src/ --subdirectory-filter src/

First, --path src/ restricts history to only commits that touch the src/ directory. Then --subdirectory-filter src/ makes that directory the new root, stripping the src/ prefix from all paths. The result is a clean repository with the exact history of src/ as if it had always been the top-level.

Example: Rewriting Commit Messages

To prepend a string to every commit message, use --message-callback:

git filter-repo --message-callback '
  return b"[MIGRATED] " + message
'

More complex logic can conditionally modify messages based on branch names or commit metadata, but the core idea remains a simple Python callback returning a byte string.

Example: Bulk Renaming Paths

If you need to rename a directory across the entire history, --path-rename is extremely handy:

git filter-repo --path-rename old-dir/:new-dir/

This changes every reference from old-dir/ to new-dir/ in all commits.

Best Practices for History Rewriting

Regardless of the tool you choose, rewriting history demands caution. Follow these practices to avoid disaster:

• Always start from a fresh clone. Never rewrite the history of your only copy of a repository. Use git clone --mirror or a regular clone and then run the rewrite inside it.
• Backup before rewriting. Keep the original repository intact until you’ve verified the rewritten history is correct.
• Test on a small branch first. Apply the filter to a single branch to confirm the output before processing all refs.
• Use --dry-run or inspect results. git filter-repo supports --dry-run to show what would happen without actually rewriting.
• Coordinate with your team. If the repository is shared, all contributors must agree to replace their history with the new rewritten version. They will need to re-clone or use git rebase carefully.
• Prefer git filter-repo for all new tasks. It’s faster, safer, and actively maintained. Leave filter-branch for legacy scripts you’ve already vetted.
• Clean up after filter-branch. If you must use it, delete the backup refs with git update-ref -d refs/original/... and then git reflog expire --expire=now --all followed by git gc --aggressive --prune=now to reclaim space.
• Force-push with care. After rewriting, pushing will require --force. Always specify the exact refs, e.g., git push origin --force --all --tags, and communicate the change to everyone.

Conclusion

git filter-branch served the Git community for over a decade, but its complexity and performance issues have made it obsolete for modern workflows. git filter-repo solves the same problems with a cleaner, faster, and more reliable design. Whether you need to excise a secret, trim repository bloat, reshape an entire codebase, or correct commit metadata, git filter-repo provides an intuitive command set that gets the job done without the pitfalls of shell-based filters. Remember that history rewriting is a destructive operation — always work on a clone, back up the original, and coordinate with collaborators. With the right tool and careful planning, you can keep your repository history clean, secure, and manageable.