Posted by Paul Wegemann on January 18, 2010 at 4:34 PM
I cross posted this on the RMNP forum.
I just a discovered some corrupt raw files that, due to my automated backup, have made they're way across all my backup drives, so I have no more good files to restore from. I use a series of hard drives for my backups, having given up DVDs quite some time ago. It seems to me that with potentially tens of thousands of files on our drives at least a few might be corrupted. How do we prevent a bad file from making its way onto a backup and possibly over-writing a previously good version of itself? I thought about having multiple iterations of backups and never overwriting, but at some point cost and space become an issue. Does anyone know of any automated data validation tools? I'd love to hear how others are handling this.
Paul
Posted by Paul Ashton on January 19, 2010 at 3:30 AM
I use SYNCTOY which is a freebie from Microsoft. I have it setup to add new and altered files but not overwrite existing files. If I delete a file from my main drive it does not get deleted from my backup. Simple and effective to use.
Posted by Gene Lugo on January 19, 2010 at 5:08 AM
Synctoy works great. I installed Syncback from Two Bright Sparks before I knew about Synctoy. Another crucial part of this is to offload your datafiles to an online backup and data warehousing service. I like Carbonite. They offer unlimited space, 100 gb/month bandwidth, and costs less than $60/yr.
Unless you are using enterprise-level (suitable for RAID) hard drives, the likelihood of data corruption increases. Not that these drives are going to eliminate the possibility getting bad files, but it should improve things a bit.
Posted by David Bostedo on January 19, 2010 at 8:43 AM
Simply put, if you want to avoid corruption altogether you have to archive to something like DVDs, or a totally separate hard drive system (online backups would qualify). Backup programs that check data will avoid errors during backup, and high end drives will lessen errors in master files. But basically, any error that sneaks into the master file will make its way to all the backups.
I know of some photographers who will make two DVD copies of their raw (to help in case the DVDs develop corruption, then have two separate hard drive based backups in their system, in addition to an occasional cloned drive that they store away from their other backups. I don't do much myself, except a basic hard drive backup. I guess it depends on how much to want to avoid file errors and data loss.
Posted by Paul Wegemann on January 19, 2010 at 2:00 PM
Discussion
4 repliesI just a discovered some corrupt raw files that, due to my automated backup, have made they're way across all my backup drives, so I have no more good files to restore from. I use a series of hard drives for my backups, having given up DVDs quite some time ago. It seems to me that with potentially tens of thousands of files on our drives at least a few might be corrupted. How do we prevent a bad file from making its way onto a backup and possibly over-writing a previously good version of itself? I thought about having multiple iterations of backups and never overwriting, but at some point cost and space become an issue. Does anyone know of any automated data validation tools? I'd love to hear how others are handling this.
Paul
Unless you are using enterprise-level (suitable for RAID) hard drives, the likelihood of data corruption increases. Not that these drives are going to eliminate the possibility getting bad files, but it should improve things a bit.
I know of some photographers who will make two DVD copies of their raw (to help in case the DVDs develop corruption, then have two separate hard drive based backups in their system, in addition to an occasional cloned drive that they store away from their other backups. I don't do much myself, except a basic hard drive backup. I guess it depends on how much to want to avoid file errors and data loss.