Posted by Paul Wegemann on January 18, 2010 at 4:31 PM
I just a discovered some corrupt raw files that, due to my automated backup, have made they're way across all my backup drives, so I have no more good files to restore from. I use a series of hard drives for my backups, having given up DVDs quite some time ago. It seems to me that with potentially tens of thousands of files on our drives at least a few might be corrupted. How do we prevent a bad file from making its way onto a backup and possibly over-writing a previously good version of itself? I thought about having multiple iterations of backups and never overwriting, but at some point cost and space become an issue. Does anyone know of any automated data validation tools? I'd love to hear how others are handling this.
Paul
Posted by Jesse Speer on January 18, 2010 at 4:45 PM
I know many here burn DVD's after every shoot - all raw files straight from the cards.
Posted by Ken Wyatt on January 18, 2010 at 6:37 PM
Great link Jesse! Certainly may come in handy some day.
Ken
Posted by Paul Wegemann on January 18, 2010 at 9:05 PM
Thanks for the link, Jesse. Unfortunately, that utility didn't work, but I'm gonna keep it on-hand anyway.
Posted by Craig Schultz on January 19, 2010 at 10:17 PM
Paul, I think this should be a function of any DAM application. They should all take a hash of the data and store it along with the other meta data. Then a validation routine could be run when you want to verify that the file has not changed/been damaged. DNG has something in the new specification for this, but nobody is using that either (at least not yet).
Unfortunately, I don't know of any DAM applications that have this capability out of the box. I use IDimager and have written my own hashing/validation routines using it's scripting capabilities. It's far from perfect and by it's nature of having to scan each file to generate the hash, it's not the fastest either. BUT, I will know if any of my files change.
BTW, this will only work for images that don't get resaved or changed when meta data it written back out to them from the DAM application (i.e. RAW images with xmp sidecars) because the hash will obviously change if the file is updated on purpose.
Craig
Posted by Paul Wegemann on January 20, 2010 at 9:16 AM
Thanks for your feedback, Craig. I'm on a quest now to figure this out. One thing I'm quickly realizing is that I'm going to need more hard drives! I gave up on DVDs awhile ago. I can't wait for the rumored 200GB Blu Ray.
Discussion
5 repliesPaul
I know many here burn DVD's after every shoot - all raw files straight from the cards.
Ken
Unfortunately, I don't know of any DAM applications that have this capability out of the box. I use IDimager and have written my own hashing/validation routines using it's scripting capabilities. It's far from perfect and by it's nature of having to scan each file to generate the hash, it's not the fastest either. BUT, I will know if any of my files change.
BTW, this will only work for images that don't get resaved or changed when meta data it written back out to them from the DAM application (i.e. RAW images with xmp sidecars) because the hash will obviously change if the file is updated on purpose.
Craig