Duplicate files

duff the duplicate file finder can find recursively such files using duff -r <directory>

A file list (skipping one of the files to be kept) can be created using duff -re <directory> > ../del.txt

After that while read file ; do rm "$file" ; done < ../del.txt will remove the files

Alternatively this can be done with rm $( duff -rPze <directory> )

A duplicate file finder for Linux file system should understand links. duff has command line options for that, as -P do not follow symbolic links to directories (symbolic links to files are never followed)

-z does not compare files with 0 size

To improve speed most programs look first for the disk space and even for the first bytes to match and if everything is identical the time consuming checksum is calculated.


Linurs Hosttech startpage