Description:

DedupeCopy 0.4.1

Find duplicates / copy and restructure file layout command-line tool.This is a simple multi-threaded file copy tool designed for consolidating andrestructuring sprawling file systems.The most common use case is for backing up data into a new layout, ignoringduplicated files.Other uses include: 1. Getting a .csv file describing all duplicated files 2. Comparing different file systems 3. Restructuring existing sets of files into different layouts (such as sorted by extension or last modification time)This tool is *NOT* a Robocopy or rsync replacement and does not try to fillthe role those play.As with all code that walks a file tree, please use with caution and expectabsolutely no warranty! :)Command examples: Generate a duplicate file report for a path: dedupecopy -p /Users/ -r dupes.csv -m manifest Copy all *.jpg files from multiple paths to a /YYYY_MM/*.jpg structure dedupecopy -p C:\pics -p D:\pics -e jpg -R jpg:mtime -c X:\pics Copy all files from two drives to a single target, preserving the path for all extensions: dedupecopy -p C:\ -p D:\ -c X:\ -m X:\manifest -R *:no_change Resume an interrupted run (assuming "-m manifest" used in prior run): dedupecopy -p /Users/ -r dupes_2.csv -i manifest -m manifest Sequentially copy different sources into the same target, not copying duplicate files (2 sources and 1 target): 1.) First record manifests for all devices dedupecopy -p \target\share -m target_manifest dedupecopy -p \source1\share -m source1_manifest dedupecopy -p \source2\share -m source2_manifest 2.) Copy each source to the target (specifying --compare so manifests from other sources are loaded but not used as part of the set to copy and --no-walk to skip re-scan of the source): dedupecopy -p \source1\share -c \target\share -i source1_manifest --compare source2_manifest --compare target_manifest --no-walk dedupecopy -p \source2\share -c \target\share -i source2_manifest --compare source1_manifest --compare target_manifest --no-walkComplete example: Assuming you start with a set of files laid out as follows: C:\ pics some_photos photo1.jpg photo2.jpg photo3.jpg D:\ pics copied_photo photo3.jpg photo4.jpg And you run the command: > dedupecopy -p C:\pics -p D:\pics -e jpg -R jpg:mtime -c X:\photos You would end up with the following (photo3.jpg was a true duplicate): X:\ photos 2012_08 photo1.jpg photo2.jpg photo3.jpg 2012_09 photo4.jpgThis project is on bitbucket: http://www.bitbucket.org/othererik/dedupe_copyThis project is on pypi: http://pypi.python.org/pypi/DedupeCopy

License

For personal and professional use. You cannot resell or redistribute these repositories in their original state.

Share

• Share on Facebook
• Share on Twitter