Using BFG Repo-Cleaner and Filter-Repo

In the world of software development, managing repositories is a crucial task. One common challenge developers face is cleaning up their repositories by removing sensitive data such as passwords, private keys, or large files that were accidentally committed. This guide will walk you through using two powerful tools: BFG Repo-Cleaner and Filter-Repo.

Introduction to BFG Repo-Cleaner

BFG Repo-Cleaner is a command-line tool designed to clean up Git repositories by removing unwanted data such as large files, private keys, or passwords. It's particularly useful for repositories with a long history where sensitive information has been committed multiple times.

Installation

To use BFG Repo-Cleaner, you first need to download the JAR file from its official GitHub repository:

wget https://repo1.maven.org/maven2/com/madgag/bfg/1.13.0/bfg-1.13.0.jar

Alternatively, you can clone the BFG Repo-Cleaner repository and build it from source.

Basic Usage

The basic command to use BFG Repo-Cleaner is:

java -jar bfg-1.13.0.jar --delete-files <file-pattern> <path-to-repo>

For example, to remove all files with the .key extension:

java -jar bfg-1.13.0.jar --delete-files *.key my-repo.git

Advanced Usage

BFG Repo-Cleaner offers several advanced options for more complex cleaning tasks. Here are a few examples:

Removing Large Files

To remove large files, you can use the --strip-blobs-bigger-than option:

java -jar bfg-1.13.0.jar --strip-blobs-bigger-than 100M my-repo.git

This command removes all blobs larger than 100MB.

Removing Sensitive Data

To remove sensitive data like passwords or API keys, you can use the --replace-text option:

java -jar bfg-1.jar --replace-text passwords.txt my-repo.git

Create a passwords.txt file with all the sensitive strings you want to replace.

Post-Cleaning Steps

After cleaning your repository with BFG Repo-Cleaner, you need to rewrite the Git history and force push the changes:

cd my-repo.git
git reflog expire --expire=now --all && git gc --prune=now --aggressive
git push origin --force --all
git push origin --force --tags

Introduction to Filter-Repo

Filter-Repo is another powerful tool for cleaning up Git repositories. It's a Python script that provides more flexibility and control over the cleaning process compared to BFG Repo-Cleaner.

Installation

To use Filter-Repo, you need to clone its repository and make it executable:

git clone https://github.com/newren/git-filter-repo.git
cd git-filter-repo
chmod +x git-filter-repo

You can then add the script to your PATH for easier access.

Basic Usage

The basic command to use Filter-Repo is:

./git-filter-repo --path <directory> --invert-paths

This command removes all files outside of the specified directory.

Advanced Usage

Filter-Repo offers a wide range of options for advanced cleaning tasks. Here are a few examples:

Removing Large Files

To remove large files, you can use the --strip-blobs-bigger-than option:

./git-filter-repo --strip-blobs-bigger-than 100M

This command removes all blobs larger than 100MB.

Removing Sensitive Data

To remove sensitive data like passwords or API keys, you can use the --replace-text option:

./git-filter-repo --replace-text passwords.txt

Create a passwords.txt file with all the sensitive strings you want to replace.

Post-Cleaning Steps

After cleaning your repository with Filter-Repo, you need to force push the changes:

git push origin --force --all
git push origin --force --tags

Best Practices

1. Backup Your Repository: Always make a backup of your original repository before performing any cleanup operations.
2. Test Locally: Perform the cleanup operations on a local copy of the repository to ensure everything works as expected.
3. Communicate with Team Members: Inform your team members about the changes, especially if you're force pushing to shared branches.
4. Monitor for Issues: After cleaning and force pushing, monitor the repository for any issues that may arise.

Conclusion

BFG Repo-Cleaner and Filter-Repo are powerful tools for cleaning up Git repositories. Each tool has its strengths and can be used depending on your specific needs. BFG Repo-Cleaner is great for quick cleanup tasks, while Filter-Repo offers more flexibility and control for complex scenarios. By following the steps outlined in this guide, you can effectively manage sensitive data and maintain a clean repository history.