Finddupe is a tool for quick detecting of duplicate files on a hard drive under Windows. Duplicate files can be detected, hardlinked, or deleted. Finddupe can also search for hardlinked files on your computer.
Finddupe is a command line program. If you aren't familiar with the "command prompt" under windows, this program might be a little challenging for you to use.
Finddupe has several possible uses:
Deleting duplicate files
When working thru somebody else's photo collection, or MP3 collection, this tool is useful for deleting the files that are duplicate. Depending on how the media is organized, there can be a lot of duplicate files in a collection.
Freeing hard drive space
Sometimes its intentional to have certain media in multiple places. By running finddupe, and hard linking the identical files, you can keep the files in multiple places, while only having one physical copy on the hard drive.
Detecting changed files for backup
Finddupe is useful for detecting which files have changed and need backing up. Simply back up the media, and then run finddupe to eliminate those files in the copy that are already contained in a previous backup.
Detecting hardlinked groups of files
Unknown to most, the winodws NTFS file system has the ability to hard link files together. If you use finddupe with the -listlink option, finddupe will search for files that appear in multiple places as hard links.
|-hardlink||Delete duplicate copies of file, and replace duplicates with hardlinks to other copy of the file. Works only on NTFS file systems, and with administrator privileges. (The C: drive under XP is almost always NTFS, and most people log in as administrator)|
|-del||Delete duplicate files|
|-sigs||Pring computed file signature of each file. The file signature is computed using a CRC of the first 32k of the file, as well as its length. The signature is used to detect files that are probably duplicates. Finddupe does a full binary file compare before taking any action.|
|-rdonly||Also operate on files that have the readonly bit set (these are normally skipped). I use this feature to eliminate shared files in large projects under version control at work.|
|-z||Do not skip zero length files (zero length files are ignored by default)|
|-u||Do not print a warning for files that cannot be read|
|-p||Hide progress indicator (useful when output is redirected to a file)|
|-j||Follow NTFS junctions and reparse points (not followed by default)|
|-bat <batchfile>||Do not hardlink or delete any files. Rather, create a batch file containing the actions to be performed. This can be useful if you want to inspect what finddupe will do.|
|-listlink||Puts finddupe in hardlink finding mode. In this mode, finddupe will list which groups of files are hardlinked together. All hardlinked instances found of a file are shown together. However, finddupe can only find instances of the hardlinked file that are within the search path. This option can only be combined with the -v option.|
|-ref <filepat>||The file or file pattern after the -ref is a reference. These files will be compared against, but not eliminated. Rather, other files on the command line will be considered duplicates of the reference files.|
|[filepat]||File pattern matching in finddupe is very powerful. It uses the same code as is used in jhead. For example, to specify c:\** would indicate every file on the entire C drive. Specifying C:\**\foo\*.jpg specifies any file that ends with .jpg that is in a subdirectory called foo anywhere on the hard drive, including such directories as c:\foo, c:\bar\foo, c:\hello\workd\foo and c:\foo\bar\foo.|
finddupe -del -ref c:\prev_backup c:\new_backup
If you have a large photo collection on c:\photos, and you wish to replace duplicates with hard links, you can run:
finddupe -hardlink c:\photosNote that this only works on NTFS file systems (such as the C drive under Windows XP). It won't work on FAT file systems, like the ones used on most external hard disks or USB flash drives.
If you want to know which files within a directory tree are hardlinked together, such as after
running the above command, you can run:
finddupe -listlink c:\photos
finddupe -bat work.bat -del c:\media\** c:\media2\**This will create the file "work bat" with file delete commands in it. The '-bat' option tells finddupe to not do anything, but rather store the actions to a batch file. This allows you to review what finddupe would do before taking any action. The '**' tells it to recursively do all the files.
Command Prompt - finddupe
C:\>finddupe testdir\* Duplicate: 'testdir\aab.txt' With: 'testdir\aab.zzz' Duplicate: 'testdir\aab.bak' With: 'testdir\aac.txt' Duplicate: 'testdir\dup1' With: 'testdir\foo2' Duplicate: 'testdir\foo' With: 'testdir\makefile' Duplicate: 'testdir\foo' With: 'testdir\makefile.bak' Duplicate: 'testdir\dup1' With: 'testdir\myglob.bak' Duplicate: 'testdir\longdiff.bak' With: 'testdir\nadine.txt' Files: 23285 kBytes in 23 files Dupes: 6971 kBytes in 7 files
I also wrote it to be very fast. For large media files, this helps a lot. Finddupe will only read the first 32k of a file and compute a hash based on that. Only if that matches with another file will it even read the entire files. I use it mostly on various media, like jpegs and mp3s to find and eliminate duplicates I may have
On modifying one file of a hardlinked pair or set, windows may not update the size of the other linked instances right away. Also, programs that move the old file to a backup copy tend to break (unlink) the hard link on saving. This behaviour may or may not be desirable.
|finddupe.exe||finddupe executable (53k)|
|finddupe-src.zip||Source code in a zip file (11k). Build with Microsoft Visual C++|
Got questions? Email me:
Other handy free utilities by Matthias Wandel:
Jhead A program for examining and manipulating digicam image metadata
To Matthias Wandel's home page