Posted by David Prichard on September 21, 2010 at 8:18 PM
Time to get more image storage. Everybody and their brother seems to have a different opinion, RAID, server, Drobo, etc.
Currently thinking of expanding to 4T and would like to do it as inexpensively, not cheaply, as possible without too much technical hassle. Any suggestions or success stories welcome.
Posted by Buck Ward on September 21, 2010 at 8:43 PM
I rely on Level 1 RAID, for redundancy, not for increased capacity. It has saved my bacon twice. A hard drive fails, but the system continues on as if nothing had happened, no data is lost and the RAID is restored when the drive is replaced. For me it is a necessity. I currently have two RAID mirrored pairs of drives in my home PC, one internal and one external.
While I see the need for mirroring, I have no interest in striping. Additional capacity can simply be achieved in independent multiple drives.
Posted by Drew Fulton on September 21, 2010 at 11:45 PM
Dave,
Great question and one that gets asked a lot. Here's my take.
RAID 0 - Striping - Useful to combine two (or more) drives to make a larger single volume. This also dramatically increases the read/write speed for this volume. The disadvantage is, if one drive goes down you lose it all.
RAID 1 - Mirroring - This type of RAID uses a pair of drives and treats it as a single drive. For example, you have two 1 TB drives, they end up being exact copies of each other and you have 1 TB of storage available. This is great because if a Hard Drive fails physically, you pull it out and put a new one in and the system restores the mirror from the other drive. The biggest problem is that this doesn't protect against digital problems, only physical ones. If you delete a file, it's gone on both drives. If a virus hits your computer, it kills both drives.
JBOD - "Just a Bunch of Drives" This approach is as it is named. It uses a bunch of drives. Dave, in your case, you are probably looking at either 3x 1.5 TB drives of 2x 2TB drives to cover your 4 TB. For backup, you have a matching set of external drives that you frequently use for backup. You still end up buying 2 drives for every 1 you need, but there is a time delay between the time something happens on the main drive and when that gets duplicated on the backup drive.
Picture each of the following "worse case scenarios" and see how each setup handles it...
1) Physical drive failure
-RAID0 - BAD, loss of all data across all drives in RAID set
-RAID1 - Replace bad drive and data is restored almost immediately (time for drives to copy), but data is not available until bad drive is replaced
-JBOD - Replace bad drive, copy data from backup over to new drive... Only data loss occurs since your last backup
2) Virus Attack or Data Corruption
RAID0 - Possible corruption/deletion of files, depends on virus/data corruption
RAID1 - Any data is immediately copied to mirror, loss of whatever is targeted by virus or corruption
JBOD - Back up drive is secure (if not connected to computer), data loss only occurs if you have backed up the infected/corrupted disk or not run a backup since loading new images.
3) Human Error - Deleted or Saved over a File
RAID0 - Out of luck, only one copy of file
RAID1 - Actions immediately mirrored to backup drive, out of luck
JBOD - If mistake caught before running next backup, can restore file from old backup
I personally use JBOD's for all my personal archives. I have a second set of drives that are backed up on a regular basis. I do have a pair of 1.5TB drives combined using in a RAID 0 set for use as a Video editing drive. Any data that resides on this drive has been previously placed on one of my Archive drives and is therefore backed up. However the RAID0 set helps provide the speed required for editing full HD AVCHD video.
The final aspect to think about is including a Right Once copy of your data, aka a DVD or Blu-Ray. If you burn yourself a set of DVDs or Blu-Rays then you have your data on a disk that can not be edited by a virus. Of course, DVDs and Blu-Rays (like hard drives) have a limited shelf life so beware of that. I personally burn 2 DVD copies, one of which gets stored out of state at my parents.
Wow, this sort of turned into a treatise on backup and archiving... I am sorry this got so long but hope it was helpful. I hope it was helpful. Please let me know if you have any more questions.
Drew
Posted by Lon Overacker on September 22, 2010 at 12:25 AM
Drew has provided an EXCELLENT summary.
IMO, and not that it matters but in my day job I work in the IT business supporting ginormous storage arrays..... Raid 1 is usually considered for performance only. If money is no object and you want to pay for twice the storage, go for it. But as Drew points out... if you corrupt, delete or otherwise muck up your original... it's get's mirrored to the other drive. out of luck.
Raid 5/6 and virtualization of multiple spindles is used for capacity and where performance is secondary (generalizing...) For home use, you'll need at least 5 spindles to make it worth your while. If a drive does fail, you're working in a degraded mode (parity re-generation) until the drive is replaced and parity data is re-written to the new drive. You are now a storage administrator.
Since PhotoShop is a memory and cpu intensive activity, having a fast disk subsystem is somewhat irrelevant. We work on ONE file at a time (ok, perhaps several for stitching, etc.,) but again that is RAM intensive.
Spend your money on multiple independent drives and a decent backup software implementation. I chose Acronis over Norton's Ghost software and perform regular daily backups. I keep two archival copies of all my image files and in addition working copies on my internal hard drive. At any given time I have at least 2, sometimes 3 copies of an image file (not including jpgs, print tif's, etc. etc.) I also now clone my boot drive so in case of a crash, virus or whatever I can boot on the secondary drive and continue on like nothing happened.
All of this without any fancy "storage servers," dedicated storage devices, etc. etc. Drives are cheap - get many and simply create your own backup system. Just make sure you develop and adhere to a backup strategy.
Regards,
Lon
Posted by Gene Lugo on September 22, 2010 at 6:34 AM
Been advising clients in my IT business for what seems like forever. Mission critical applications require 0% downtime, so RAIDed storage servers are the defacto standard. I would never suggest anyone implement a RAID array without a parity drive (or two). Typically you would use RAID 6 with 12 or more drives, 5 with 10 or less. Advantage of 6 over 5 is that you will usually implement this with a drive controller with serious hardware acceleration. The advantage is that you can suffer 2 drive failures without taking down the array. RAID 5 is more cost effective but only has a single parity drive, so if you lose a drive and are in a rebuild state and another drive faults, you will lose data.
An often made mistake is to use the cheapest OEM drives to build an array. This is never a good idea. Those 2 TB drives selling for $99 are not your friend. Their data integrity and physical/mechanical failure rate is too high for RAID. Drive manufacturers offer "enterprise" drives that are designed and built to higher standards than the run-of-the-mill special, and this is what should be used.
Now the decision to use RAID has to do with hardware fault tolerance - not backup. There is no substitute for a solid backup strategy. On site data redundancy along with offsite storage, such as with Carbonite, is a manageable solution.
Posted by Walt Sterneman on September 22, 2010 at 2:22 PM
Wow, this is all great info. I am really glad you posted this David.
I just had a 1.5T drive fail. I had an offsite drive backup that was easy to get most of my files back from. I also use Carbonite and was able to get the files I'd saved since the last offsite backup from there. You do not want to depend fully on Carbonite as their download speed is ridiculously slow.
At any rate as a result I was thinking about RAID, but I have now been talked out of it.
I can atest that a RAID-0, while being fast, is not safe. I had a RAID-0 on my machine, had a drive go bonkers, and I lost everything. Fortunately, I had all my images and other documents, including my Outook pst file, backed up, so these were safe.
I considered getting something a Drobo with a RAID-1, but decided to go with one external, one internal, and an internal drive on another (networked) machine for backup. My strategy appears to be working....so far.
--P
Posted by Buck Ward on September 22, 2010 at 7:17 PM
In my post above, I said I use Level 1 RAID (mirroring). This is for redundancy, not for back-up. I also automatically back up to a separate drive daily, with a separate automatic back-up weekly. If I have a problem with a file, I can retrieve it because It backed up yesterday. If I don't discover it before the corrupted or lost file is also backed up, I have a copy of it on last weeks back-up. RAID can be used in a back-up strategy, but it isn't a back-up strategy itself.
I agree with redundancy: I just can't afford it. A RAID-1 is a good way to go without spending a small fourtune. I also agree that RAID is not designed as a backup strategy--it is designed for either speed (RAID-0) or redundancy.
--P
Posted by David Prichard on September 22, 2010 at 9:52 PM
Thanks all for the information. It does, of course, raise more questions. I had been planning to upgrade to Windows 7 64 bit, I know, get a Mac , largely for Lightroom and Photoshop, but was holding off until more 64 bit software became available.
The current machine does it all, internet, office and photo with a 1T internal for image storage and a 750G external backup. As I had planned to upgrade the machine when more 64 bit software was available would it make sense to spend the extra for a dedicated photo machine with a RAID 1 array?
I can't proffer and opinion on the new machine Dave, but I will say that you folks really sent a chill up my spine!!
In recent weeks I've accumulated a large number of exceptional bear photos from the best shooting ops I've experienced in 30 years. But we've also been really busy overall.
You made me realize that all those images weren't backed up in any way!!!!
The panic has passed. They're now on three separate drives in addition to my computer, and I've burned two sets of DVDs. But I won't relax completely till one of the latter is housed in our safe deposit box later today!
Thanks for putting the boot where it belongs!
Posted by Ed Cordes on September 23, 2010 at 2:05 PM
I use JBOD. At the end of each session I back up to an external eSATA 2 TB drive using SyncToy. Then about once a month I back that external up to an additional external which is stored at work. As I fill up the 2 TB drives I simply add more retiring the older one to the off site archive and get a new one for on site access of the older data. the key to making this work is discipline to back up each session. However I only back up when I am sure there is no corruption of data as mentioned above. I do not trust RAID in my set up as it is only a redundancy application not a true back up.
Discussion
11 repliesCurrently thinking of expanding to 4T and would like to do it as inexpensively, not cheaply, as possible without too much technical hassle. Any suggestions or success stories welcome.
While I see the need for mirroring, I have no interest in striping. Additional capacity can simply be achieved in independent multiple drives.
Great question and one that gets asked a lot. Here's my take.
RAID 0 - Striping - Useful to combine two (or more) drives to make a larger single volume. This also dramatically increases the read/write speed for this volume. The disadvantage is, if one drive goes down you lose it all.
RAID 1 - Mirroring - This type of RAID uses a pair of drives and treats it as a single drive. For example, you have two 1 TB drives, they end up being exact copies of each other and you have 1 TB of storage available. This is great because if a Hard Drive fails physically, you pull it out and put a new one in and the system restores the mirror from the other drive. The biggest problem is that this doesn't protect against digital problems, only physical ones. If you delete a file, it's gone on both drives. If a virus hits your computer, it kills both drives.
JBOD - "Just a Bunch of Drives" This approach is as it is named. It uses a bunch of drives. Dave, in your case, you are probably looking at either 3x 1.5 TB drives of 2x 2TB drives to cover your 4 TB. For backup, you have a matching set of external drives that you frequently use for backup. You still end up buying 2 drives for every 1 you need, but there is a time delay between the time something happens on the main drive and when that gets duplicated on the backup drive.
Picture each of the following "worse case scenarios" and see how each setup handles it...
1) Physical drive failure
-RAID0 - BAD, loss of all data across all drives in RAID set
-RAID1 - Replace bad drive and data is restored almost immediately (time for drives to copy), but data is not available until bad drive is replaced
-JBOD - Replace bad drive, copy data from backup over to new drive... Only data loss occurs since your last backup
2) Virus Attack or Data Corruption
RAID0 - Possible corruption/deletion of files, depends on virus/data corruption
RAID1 - Any data is immediately copied to mirror, loss of whatever is targeted by virus or corruption
JBOD - Back up drive is secure (if not connected to computer), data loss only occurs if you have backed up the infected/corrupted disk or not run a backup since loading new images.
3) Human Error - Deleted or Saved over a File
RAID0 - Out of luck, only one copy of file
RAID1 - Actions immediately mirrored to backup drive, out of luck
JBOD - If mistake caught before running next backup, can restore file from old backup
I personally use JBOD's for all my personal archives. I have a second set of drives that are backed up on a regular basis. I do have a pair of 1.5TB drives combined using in a RAID 0 set for use as a Video editing drive. Any data that resides on this drive has been previously placed on one of my Archive drives and is therefore backed up. However the RAID0 set helps provide the speed required for editing full HD AVCHD video.
The final aspect to think about is including a Right Once copy of your data, aka a DVD or Blu-Ray. If you burn yourself a set of DVDs or Blu-Rays then you have your data on a disk that can not be edited by a virus. Of course, DVDs and Blu-Rays (like hard drives) have a limited shelf life so beware of that. I personally burn 2 DVD copies, one of which gets stored out of state at my parents.
Wow, this sort of turned into a treatise on backup and archiving... I am sorry this got so long but hope it was helpful. I hope it was helpful. Please let me know if you have any more questions.
Drew
IMO, and not that it matters but in my day job I work in the IT business supporting ginormous storage arrays..... Raid 1 is usually considered for performance only. If money is no object and you want to pay for twice the storage, go for it. But as Drew points out... if you corrupt, delete or otherwise muck up your original... it's get's mirrored to the other drive. out of luck.
Raid 5/6 and virtualization of multiple spindles is used for capacity and where performance is secondary (generalizing...) For home use, you'll need at least 5 spindles to make it worth your while. If a drive does fail, you're working in a degraded mode (parity re-generation) until the drive is replaced and parity data is re-written to the new drive. You are now a storage administrator.
Since PhotoShop is a memory and cpu intensive activity, having a fast disk subsystem is somewhat irrelevant. We work on ONE file at a time (ok, perhaps several for stitching, etc.,) but again that is RAM intensive.
Spend your money on multiple independent drives and a decent backup software implementation. I chose Acronis over Norton's Ghost software and perform regular daily backups. I keep two archival copies of all my image files and in addition working copies on my internal hard drive. At any given time I have at least 2, sometimes 3 copies of an image file (not including jpgs, print tif's, etc. etc.) I also now clone my boot drive so in case of a crash, virus or whatever I can boot on the secondary drive and continue on like nothing happened.
All of this without any fancy "storage servers," dedicated storage devices, etc. etc. Drives are cheap - get many and simply create your own backup system. Just make sure you develop and adhere to a backup strategy.
Regards,
Lon
An often made mistake is to use the cheapest OEM drives to build an array. This is never a good idea. Those 2 TB drives selling for $99 are not your friend. Their data integrity and physical/mechanical failure rate is too high for RAID. Drive manufacturers offer "enterprise" drives that are designed and built to higher standards than the run-of-the-mill special, and this is what should be used.
Now the decision to use RAID has to do with hardware fault tolerance - not backup. There is no substitute for a solid backup strategy. On site data redundancy along with offsite storage, such as with Carbonite, is a manageable solution.
I just had a 1.5T drive fail. I had an offsite drive backup that was easy to get most of my files back from. I also use Carbonite and was able to get the files I'd saved since the last offsite backup from there. You do not want to depend fully on Carbonite as their download speed is ridiculously slow.
At any rate as a result I was thinking about RAID, but I have now been talked out of it.
I considered getting something a Drobo with a RAID-1, but decided to go with one external, one internal, and an internal drive on another (networked) machine for backup. My strategy appears to be working....so far.
--P
I agree with redundancy: I just can't afford it. A RAID-1 is a good way to go without spending a small fourtune. I also agree that RAID is not designed as a backup strategy--it is designed for either speed (RAID-0) or redundancy.
--P
The current machine does it all, internet, office and photo with a 1T internal for image storage and a 750G external backup. As I had planned to upgrade the machine when more 64 bit software was available would it make sense to spend the extra for a dedicated photo machine with a RAID 1 array?
In recent weeks I've accumulated a large number of exceptional bear photos from the best shooting ops I've experienced in 30 years. But we've also been really busy overall.
You made me realize that all those images weren't backed up in any way!!!!
The panic has passed. They're now on three separate drives in addition to my computer, and I've burned two sets of DVDs. But I won't relax completely till one of the latter is housed in our safe deposit box later today!
Thanks for putting the boot where it belongs!