Analysis Options¶

--analyze

Filtering based on paths (see also --filename-callback)¶

These options specify the paths to select. Note that much like git itself, renames are NOT followed so you may need to specify multiple paths, e.g. --path olddir/ --path newdir/

--invert-paths

--path-match <dir_or_file>, --path <dir_or_file>

--path-glob <glob>

--path-regex <regex>

--use-base-name

Renaming based on paths (see also --filename-callback)¶

Note: if you combine path filtering with path renaming, be aware that a rename directive does not select paths, it only says how to rename paths that are selected with the filters.

--path-rename <old_name:new_name>, --path-rename-match <old_name:new_name>

Path shortcuts¶

--paths-from-file <filename>

--subdirectory-filter <directory>

--to-subdirectory-filter <directory>

Content editing filters (see also --blob-callback)¶

--replace-text <expressions_file>

--strip-blobs-bigger-than <size>

--strip-blobs-with-ids <blob_id_filename>

Renaming of refs (see also --refname-callback)¶

--tag-rename <old:new>

Filtering of commit messages (see also --message-callback)¶

--replace-message <expressions_file>

--preserve-commit-hashes

--preserve-commit-encoding

Filtering of names & emails (see also --name-callback and --email-callback)¶

--mailmap <filename>

--use-mailmap

Parent rewriting¶

--replace-refs {delete-no-add, delete-and-add, update-no-add, update-or-add, update-and-add, old-default}

--prune-empty {always, auto, never}

--prune-degenerate {always, auto, never}

--no-ff

Generic callback code snippets¶

--filename-callback <function_body>

--message-callback <function_body>

--name-callback <function_body>

--email-callback <function_body>

--refname-callback <function_body>

--file-info-callback <function_body>

--blob-callback <function_body>

--commit-callback <function_body>

--tag-callback <function_body>

--reset-callback <function_body>

Sensitive Data Removal¶

--sensitive-data-removal, --sdr

Note that if you have any local-only changes (i.e. un-pushed
changes) in your repository, on any branch or ref, this fetch step
may discard them.  Working in a fresh clone avoids this problem;
see also the --no-fetch option if you don't want to work with a
fresh clone and you have important local-only changes.

--no-fetch

Location to filter from/to¶

--source <source>

--target <target>

Miscellaneous options¶

--help, -h

--force, -f

--partial

--refs <refs+>

--dry-run

--debug

--stdin

--quiet

Commit map¶

The $GIT_DIR/filter-repo/commit-map file contains a mapping of how all commits were (or were not) changed.

Reference map¶

The $GIT_DIR/filter-repo/ref-map file contains a mapping of which local references were (or were not) changed.

Changed References¶

The $GIT_DIR/filter-repo/changed-refs file contains a list of refs that were changed.

First Changed Commits¶

The $GIT_DIR/filter-repo/first-changed-commits contains a list of the first commit(s) changed by the filtering operation. These are the commits that got rewritten and which had no parents that were also rewritten.

So, for example if you had commits A1-B1-C1-D1-E1 before running git-filter-repo, and afterward you had commits A1-B2-C2-D2-E2 then the First Changed Commits file would contain just one line, which would be the hash of B2.

In most cases, there will only be one commit listed, but if you had multiple root commits or a non-linear history where the commits on those diverging histories were the first ones modified, then there could be multiple first changed commits and they will each be listed on separate lines.

Already Ran¶

The $GIT_DIR/filter-repo/already_ran file contains a file recording that git-filter-repo has been run. When this file is present, future runs will be treated as an extension of the previous filtering operation.

Concretely, this means: * The "Fresh Clone" check is bypassed

This is done because past runs would cause the repository to no longer
look like a fresh clone, and thus fail the fresh clone check, but doing
filtering via multiple invocations of git-filter-repo is an intended
and support usecase.  You already passed or bypassed the "Fresh Clone"
check on your initial run.

In other words, if the first filter-repo invocation rewrote commit
A to commit B, and the second filter-repo invocation rewrite
commit B to commit C, then the second run would have an "A C"
entry rather than a "B C" entry for the changed commit.

In more detail, if the repository original had the following commits:


   A1-B1-C1-D1-E1
and the first invocation of filter-repo changed this to


   A1-B1-C2-D2-E2
then the first run would report "C1" as the first changed commit.  If
a second filter-repo run further changed this to


   A1-B1-C2-D3-E3
then it would report "C1" as the first changed commit, not "D2",
because it is comparing to the original commits rather than the
intermediate ones.

However, if the already_ran file exists but is older than 1 day when they invoke git-filter-repo, the user will be prompted for whether the new run should be considered a continuation of the previous run. If they do not answer in the affirmative, then the above three bullets will not apply. This prompt exists because users might do a history rewrite in a repository, forget about it and leave the $GIT_DIR/filter-repo directory around, and then some months or years later need to do another rewrite. If commits have been made public and shared from the previous rewrite, then the next filter-repo run should not be considered a continuation of the previous filtering run.

Original LFS Objects¶

When running with the --sensitive-data-removal flag, and LFS is in use by the repository, the $GIT_DIR/filter-repo/original_lfs_objects contains a list of LFS objects referenced by the repository before the rewrite, in sorted order.

Orphaned LFS Objects¶

When running with the --sensitive-data-removal flag, and LFS is in use by the repository, the $GIT_DIR/filter-repo/orphaned_lfs_objects contains a list of LFS objects that used to be referenced by the repository but no longer are after git-filter-repo has run. Objects appear in sorted order.

Why is my origin removed?¶

When you rewrite history, all commit IDs (starting with the first one where changes are made) are modified. Even if you think you didn’t change an intermediate commit, the fact that you changed any of its ancestors is also a change that counts and will cause a commit’s ID to change as well. It is unfortunately all-too-easy for yourself or someone else to accidentally merge the old ugly history you were trying to rewrite with the new history, resulting in not only the old ugly history returning but getting you "two copies" of each commit (both an original commit and a cleaned-up alternative), and thus doubling the number of commits in your repository. In short, you end up with an even bigger mess to clean up than you started with.

This happens frequently to people using git filter-branch or BFG repo cleaner, and can happen to folks using git filter-repo if they insist on pushing back to the original repo. Example ways you can get such an even uglier history include:

Removing the origin remote and suggesting people push to a new repo (and ensuring they tell others to clone the new repo) is usually a good forcing function to avoid these problems. But, if people really want to push to the original repository despite these warnings, it is trivial to do so; simply run:

and then you can push (e.g. git push --force --branches --tags --prune). Since removing the origin url is such a cheap way to potentially prevent big messes, and it’s so easy to work around for those that really do want to push back over the original history, removing the origin url is a great safety measure that I employ.

One final warning if you really want to push back to the original repo: see the next section on sensitive data removals. Those are the steps needed when pushing back to the original repo; they are so involved that I assume they are only worth it when sensitive data is involved, but you can choose to follow them for other kinds of rewrites too.

Sensitive Data Removals¶

Sensitive data removals are a specialized type of history rewrite. While it is always very problematic to mix the cleaned-up history with the non-cleaned-up history, for sensitive data removals it is also bad to allow others to continue to view/clone/fetch the non-cleaned-up history at all; users often need to try to expunge the old history as well.

Note that if the sensitive data under consideration is a token/password/credential/secret (as is often the case), then it is important that you revoke and rotate that credential first. Once the credential is revoked or rotated, it can no longer be used for access. Revoking/rotating may resolve your problem without resorting to the heavy-handed action of rewriting and purging history.

For sensitive data removal history rewrites, there are three high-level steps:

Each will be discussed in greater detail below.

One important thing to note, though, is that others working on the same repository should be instructed to stop while you do the cleanup; if they continue development during your cleanup, you’ll likely be forced to either discard their changes or start over on your cleanup.

Rewrite the repository locally, using git-filter-repo

git log --all --name-status -- ${PROBLEMATIC_FILE1} ${PROBLEMATIC_FILE2}

git log -S"${PROBLEMATIC_STRING}" --all -p --

Make sure other copies are cleaned up: primary server

git push --force --mirror origin

Make sure other copies are cleaned up: clones of colleagues

git cat-file -t ${HASH_OF_FIRST_CHANGED_COMMIT}

git for-each-ref --contains ${HASH_OF_FIRST_CHANGED_COMMIT}

Prevent repeats and avoid future sensitive data spills

Path based filtering¶

To only keep the README.md file plus the directories guides and tools/releases/:

git filter-repo --path README.md --path guides/ --path tools/releases

Directory names can be given with or without a trailing slash, and all filenames are relative to the toplevel of the repo. To keep all files except these paths, just add --invert-paths:

git filter-repo --path README.md --path guides/ --path tools/releases --invert-paths

If you want to have both an inclusion filter and an exclusion filter, just run filter-repo multiple times. For example, to keep the src/main subdirectory but exclude files under src/main named data, run:

git filter-repo --path src/main/
git filter-repo --path-glob 'src/*/data' --invert-paths

Note that the asterisk (*) will match across multiple directories, so the second command would remove e.g. src/main/org/whatever/data. Also, the second command by itself would also remove e.g. src/not-main/foo/data, but since src/not-main/ was removed by the first command, that’s not an issue. Also, the use of quotes around the asterisk is sometimes important to avoid glob expansion by the shell.

You can also select paths by regular expression (see https://docs.python.org/3/library/re.html#regular-expression-syntax). For example, to only include files from the repo whose name is in the format YYYY-MM-DD.txt and is found at least two subdirectories deep:

git filter-repo --path-regex '^.*/.*/[0-9]{4}-[0-9]{2}-[0-9]{2}.txt$'

If you want two directories to be renamed (and maybe merged if both are renamed to the same location), use --path-rename; for example, to rename both cmds/ and src/scripts/ to tools/:

git filter-repo --path-rename cmds:tools --path-rename src/scripts/:tools/

As with --path, directories can be specified with or without a trailing slash for --path-rename.

If you do a --path-rename to something that was already in use, it will be silently overwritten. However, if you try to rename multiple files to the same location (e.g. src/scripts/run_release.sh and cmds/run_release.sh both existed and had different content with the renames above), then you will be given an error. If you have such a case, you may want to add another rename command to move one of the paths somewhere else where it won’t collide:

git filter-repo --path-rename cmds/run_release.sh:tools/do_release.sh \


                --path-rename cmds/:tools/ \


                --path-rename src/scripts/:tools/

Also, --path-rename brings up ordering issues; all path arguments are applied in order. Thus, a command like

git filter-repo --path-rename sources/:src/main/ --path src/main/

would make sense but reversing the two arguments would not (src/main/ is created by the rename so reversing the two would give you an empty repo). Also, note that the rename of cmds/run_release.sh a couple examples ago was done before the other renames.

Note that path renaming does not do path filtering, thus the following command

git filter-repo --path src/main/ --path-rename tools/:scripts/

would not result in the tools or scripts directories being present, because the single filter selected only src/main/. It’s likely that you would instead want to run:

git filter-repo --path src/main/ --path tools/ --path-rename tools/:scripts/

If you prefer to filter based solely on basename, use the --use-base-name flag (though this is incompatible with --path-rename). For example, to only include README.md and Makefile files from any directory:

git filter-repo --use-base-name --path README.md --path Makefile

If you wanted to delete all .DS_Store files in any directory, you could either use:

git filter-repo --invert-paths --path '.DS_Store' --use-base-name

or

git filter-repo --invert-paths --path-glob '*/.DS_Store' --path '.DS_Store'

(the --path-glob isn’t sufficient by itself as it might miss a toplevel .DS_Store file; further while something like --path-glob '*.DS_Store' would workaround that problem it would also grab files named foo.DS_Store or bar/baz.DS_Store)

Finally, see also the --filename-callback from the section called “CALLBACKS”.

Filtering based on many paths¶

If you have a long list of files, directories, globs, or regular expressions to filter on, you can stick them in a file and use --paths-from-file; for example, with a file named stuff-i-want.txt with contents of

# Blank lines and comment lines are ignored.
# Examples similar to --path:
README.md
guides/
tools/releases
# An example that is like --path-glob:
glob:*.py
# An example that is like --path-regex:
regex:^.*/.*/[0-9]{4}-[0-9]{2}-[0-9]{2}.txt$
# An example of renaming a path
tools/==>scripts/
# An example of using a regex to rename a path
regex:(.*)/([^/]*)/([^/]*)\.text$==>\2/\1/\3.txt

then you could run

git filter-repo --paths-from-file stuff-i-want.txt

to get a repo containing only the toplevel README.md file, the guides/ and tools/releases/ directories, all python files, files whose name was of the form YYYY-MM-DD.txt at least two subdirectories deep, and would rename tools/ to scripts/ and rename files like foo/bar/baz.text to bar/foo/baz.txt. Note the special line prefixes of glob: and regex: and the special string ==> denoting renames.

Sometimes you have a way of easily generating all the files you want. For example, if you know that none of the currently tracked files have any newlines or special characters in them (see core.quotePath from git config --help) so that git ls-files would print all files literally one per line, and you knew that you wanted to keep only the files that are currently tracked (thus deleting from all commits in history any files that only appear on other branches or that only appear in older commits), then you could use a pair of commands such as

git ls-files >../paths-i-want.txt
git filter-repo --paths-from-file ../paths-i-want.txt

Similarly, you could use --paths-from-file to delete many files. For example, you could run git filter-repo --analyze to get reports, look in one such as .git/filter-repo/analysis/path-deleted-sizes.txt and copy all the filenames into a file such as /tmp/files-i-dont-want-anymore.txt and then run

git filter-repo --invert-paths --paths-from-file /tmp/files-i-dont-want-anymore.txt

to delete them all.

Directory based shortcuts¶

Let’s say you had a directory structure like the following:

module/


   foo.c


   bar.c
otherDir/


   blah.config


   stuff.txt
zebra.jpg

If you wanted just the module/ directory and you wanted it to become the new root so that your new directory structure looked like

foo.c
bar.c

then you could run:

git filter-repo --subdirectory-filter module/

If you wanted all the files from the original repo, but wanted to move everything under a subdirectory named my-module/, so that your new directory structure looked like

my-module/


   module/


      foo.c


      bar.c


   otherDir/


      blah.config


      stuff.txt


   zebra.jpg

then you would instead run run

git filter-repo --to-subdirectory-filter my-module/

Content based filtering¶

If you want to filter out all files bigger than a certain size, you can use --strip-blobs-bigger-than with some size (K, M, and G suffixes are recognized), e.g.:

git filter-repo --strip-blobs-bigger-than 10M

If you want to strip out all files with specified git object ids (hashes), list the hashes in a file and run

git filter-repo --strip-blobs-with-ids FILE_WITH_GIT_BLOB_IDS

If you want to modify file contents, you can do so based on a list of expressions in a file, one per line. For example, with a file named expressions.txt containing

p455w0rd
foo==>bar
glob:*666*==>
regex:\bdriver\b==>pilot
literal:MM/DD/YYYY==>YYYY-MM-DD
regex:([0-9]{2})/([0-9]{2})/([0-9]{4})==>\3-\1-\2

then running

git filter-repo --replace-text expressions.txt

will go through and replace p455w0rd with ***REMOVED***, foo with bar, any line containing 666 with a blank line, the word driver with pilot (but not if it has letters before or after; e.g. drivers will be unmodified), replace the exact text MM/DD/YYYY with YYYY-MM-DD and replace date strings of the form MM/DD/YYYY with ones of the form YYYY-MM-DD. In the expressions file, there are a few things to note:

See also the --blob-callback from the section called “CALLBACKS”.

Updating commit/tag messages¶

If you want to modify commit or tag messages, you can do so with the same syntax as --replace-text, explained above. For example, with a file named expressions.txt containing

foo==>bar

then running

git filter-repo --replace-message expressions.txt

will replace foo in commit or tag messages with bar.

See also the --message-callback from the section called “CALLBACKS”.

Refname based filtering¶

To rename tags, use --tag-rename, e.g.:

git filter-repo --tag-rename foo:bar

This will rename any tags starting with foo to now start with bar. Either side of the colon could be blank, e.g.

git filter-repo --tag-rename '':'my-module-'

For more general refname modification, see --refname-callback from the section called “CALLBACKS”.

User and email based filtering¶

To modify username and emails of commits, you can create a mailmap file in the format accepted by git-shortlog(1). For example, if you have a file named my-mailmap you can run

git filter-repo --mailmap my-mailmap

and if the current contents of that file are as follows (if the specified mailmap file is version controlled, historical versions of the file are ignored):

Name For User <email@addre.ss>
<new@ema.il> <old1@ema.il>
New Name And <new@ema.il> <old2@ema.il>
New Name And <new@ema.il> Old Name And <old3@ema.il>

then we can update username and/or emails based on the specified mapping.

See also the --name-callback and --email-callback from the section called “CALLBACKS”.

Parent rewriting¶

To replace $commit_A with $commit_B (e.g. make all commits which had $commit_A as a parent instead have $commit_B for that parent), and rewrite history to make it permanent:

git replace $commit_A $commit_B
git filter-repo --proceed

To create a new commit with the same contents as $commit_A except with different parent(s) and then replace $commit_A with the new commit, and rewrite history to make it permanent:

git replace --graft $commit_A $new_parent_or_parents
git filter-repo --proceed

The --proceed option is needed to avoid failing the "no arguments specified" check. Note that older versions of git-filter-repo required --force to be passed after creating a graft to avoid triggering the not-a-fresh-clone check; that check has been modified to remove this overuse of --force.

Partial history rewrites¶

To rewrite the history on just one branch (which may cause it to no longer share any common history with other branches), use --refs. For example, to remove a file named extraneous.txt from the master branch:

git filter-repo --invert-paths --path extraneous.txt --refs master

To rewrite just some recent commits:

git filter-repo --invert-paths --path extraneous.txt --refs master~3..master

How filter-repo works¶

Roughly, filter-repo works by running

git fast-export <options> | filter | git fast-import <options>

where filter-repo not only launches the whole pipeline but also serves as the filter in the middle. However, filter-repo does a few additional things on top in order to make it into a well-rounded filtering tool. A sequence that more accurately reflects what filter-repo runs is:

Some notes or exceptions on each of the above:

Information about these steps is printed out when --debug is passed to filter-repo. When doing a --partial history rewrite, steps 2, 3, 7, and 8 are unconditionally skipped, step 5 is skipped if --replace-refs is update-no-add, and just the nuke-unused-refs portion of step 5 is skipped if --replace-refs is something else.

Limitations¶

Inherited limitations

Intrinsic limitations

Issues specific to filter-repo

Comments on reversibility