Directory Traversal and Duplicate File Deletion using the SHA-256 Hash

PURGETREE - On Windows systems, rapidly identifies duplicate files in an entire directory tree using the SHA-256 hash algorithm, then creates and executes a batch file to rapidly delete all but one of the duplicates.

USAGE:

cd('dirname'); purgetree
OR
cd('dirname');purgetree('displayonly')

dirname = The name of the highest ("parent") directory in the directory tree within which you wish to purge duplicate files.

Notes:

This function rapidly compares large numbers of files for identical content by computing the SHA-256 hash of each file and detecting duplicates. The probablility of two non-identical files having the same hash, even in a hypothetical directory containing millions of files, is exceedingly remote. Since hashes rather than file contents are compared, the process of detecting duplicates is greatly accelerated.

If PURGETREE is called with any arguments, for example purgetree('displayonly') or simply purgetree(1), then the file deletion commands are displayed in the Matlab window but NOT executed (i.e., no files are deleted.)

You must have the freeware file sha256deep.exe on your Matlab path to use this function. The sha256deep.exe file is included in the download, but is also available as part of the md5deep family of functions at:
http://md5deep.sourceforge.net/
The SHA-256 hash algorithm was developed by the National Institute of Standards and Technology (NIST) and is described at:
http://csrc.nist.gov/publications/fips/fips180-2/fips180-2.pdf

When duplicate files are found, the file that has the first alphabetically sorted name is retained; others are deleted.

Carefully tested, but no warranty; use at your own risk.

Michael Kleder, Sep 2005