Handling all these megapixels?

Like many of you, I'm drooling over the new 5D MKII, but I'm seeing these new high megapixel cameras as double edge swords. When I look at my current 2 year old pc with "only" 750GB of usable space, and my ability to back that up, I cringe at the thought of having to upgrade my entire system. I also see a severe impact on the performance of PS and Lightroom. So, especially for those of you shooting with high megapixel cameras, how do you handle all this data?

Yeah, Paul, I just posted something similar on Todd's post. No way can I afford all the upgrades necessary for the new camera. This purchase will affect almost everyone's systems. I've only got 250GB and since I got my 5d 4 months ago, more than half of that disappeared!

I'm using a 1ds3. I download my cards to an 80GB external drive (Epson P4000). On extended trips when the P40000 isn't large enough, I also have 2 more Western Digital 160GB external drives to act as mirrors to each other until I put them onto the desktop machine. I then connect the P4000 to my desktop and bring up DPP. I review the raw images using DPP. Save the ones I like to my 250GB C: drive on the desktop under various folders depending on subject matter. After reviewing again and touching up color balance, exposure etc in Adobe Camera RAW I use the batch processing of PS3's Bridge to attach copyright and other file info to all the images. I then transfer a copy of each image to two 500GB external hard drives (1 serves as a mirror to the other). Once both externals have a confirmed copy, I delete all the images that were on the desktop C: drive, thus freeing up space for the next shoot.

I tried Lightroom 1.1 but thought it was too confusing in terms of how it kept track of where everything was. Also when I disconnected the external hard drives, suddenly LR of course had no idea where any of the images were. I turn off the externals when I know I won't be accessing them to prolong their lives. I love LR's interface and large preview window and the fast manner in which you can handle images, but the catalog functions just don't mesh with how I work.

Just out of curiosity, Don, what's a ball park figure for your storage system, both in-house and on the road?

The larger files, and the impact they create, make me think more about my shooting/workflow - and not as much about computer/software. The file size is one factor in a deliberate choice I recently made to simplify my entire workflow, from shooting to finished file. The actual impact on my computer/software was not that severe, really. I am using a 5-year old computer though (although it is packed with 6GB RAM). It's kinda slow to start with. Just prepare for a little drop in your workflow speed, and you'll be OK. I will say that if you get into two or more layers in PS, it WILL dramatically slow your system down - at least in my situation. That's why I try to shoot better, and do less in PS these days.

Shoot less. Seriously. I carry six 8GB cards in my bag. I've never used all of them, up to 5 days of shooting. Start thinking more about why you shoot an image in the field - and how likely the image is going to be kept. And cut down on the amount of multiple comps of one scene. It's a way of thinking that can not only help lessen the impact on your digital darkroom, but also benefit your photography as a whole - as you inject more thought into your process.

And along the same lines, REALLY learn how to edit your own work - as in sorting out the keepers and trash files once you get home. I'm fairly harsh actually. I throw away plenty of keepers - esp. if they are duplicate comps of the same scene. You may not have a relatively large image collection (overrated anyway), but the overall quality will be better.

Two aspects of my approach anyway ... an approach that doesn't require modifying your physical setup too much.

I use a Drobo, with 2TB, for data storage (and Amazon's S3 online storage for off-site backup - slow but easier than a box of disks). The Drobo unit isn't cheap, but the internal drives are. And swapping drives as my collection grows is a snap. A solid/scalable storage solution was first on my list ... then new lenses ... and then a new camera body.

Its expensive, but luckily the price of storage, both flash and disk has been declining quickly over the last few years.

For myself, I'm facing a need to delete unneeded photos. I've got alot on my plate right now, so that maintenance project may need to wait for a while, but I'll need to accomplish it this winter. I guess I'll have to decide whether to remove those files from my backups or not.

As for individual file size, I'm cranking 4X5 scans through my e6300 PC with 3GB of ram. So far its working pretty well.

The cost of storage has gone down considerably, thank god. You can get 500gb internal or external drive for around $70-80.
I download the files using DPP on my 160gb C drive, go through them and delete what I don't like, then I move everything to a seperate internal drive just for my photos. I back that drive up unto an external drive. At that point I delete the photos on my c: to clear it up.

I will have to look into a stand alone drive like the Epson for longer trips where I can't get access to my laptop to download my cards.

Storage is relatively cheap if your computer has the expansion capability, either internal or USB ports for external.
I'm currently migrating my files and programs from my 3yr old PC to my new Mac Pro. The old PC had approx 940GB across 4 drives (320, 250, 250, 120) internal, an external 320GB drive, a Drobo with 1.2TB of storage, and an external 1TB WD Mybook. Not all of the storage is devoted to photography but a large part of it. My main working portfolio is on one of the 250GB internal drives. I back those photos and the LR catalog up to the Drobo, which provides RAID like protection. I then also back the photos and catalogs to the 1TB mybook which goes into a safe deposit box offsite.

The Mac pro came with a 320GB drive with the OS and baseline apps installed. I added a 1TB seagate drive for $140 from newegg.com. The Mac pro has 5 drive bays, so I have internal expantion for 3 more drives. As I migrate from the PC, I'll take the largest drives and plug them in and I can migrate to larger drives if needed.

Right now I pretty much keep all of my images, keepers or not. I have one collection that has everything and another that is my serious portfolio that I weed out. Still, I could afford to delete more since I don't seem to go back and revisit images like I think I will. Plus I'm keeping many more than are worthy of any attention.

The biggest impact due to image file size is processing time/power. My 4x5 scans are 100mb+, add couple of layers and things really start dragging on my old PC. Lately I've noticed severe slowdowns in LR2 when using the adjustment brush. This was one of the things that pushed me over the upgrade edge to the Mac pro. I haven't done any photo editing on the Mac yet, but with 8 2.8Ghz cores, I have yet to see it break a sweat yet. Ripping music makes the cpu meter light up a little but everything still buzzes right along. It appears to be a real workhorse, but the cost may be hard to swallow. But the power will be a big help as file sizes grow.

My approach is right in line with Jesse's. Shoot less, but more thoughtfully. I definitely don't want to sit at the PC and pour over hundreds of mediocre images from an outing. I'd rather spend that time out in the field making one or two more compelling ones.

I also don't shoot with the DSLR unless I feel there's something very compelling there. I have a 8mp Olympus p&s that does a very nice job for those mid-day documentary type shots; the DSLR comes out at sunrise, sunset, or when mid-day lighting is very compelling (ie., storms).

If I do take duplicate images of a comp, I figure out which is most compelling, work it up in PS, make sure I'm happy with it, and delete the rest. Having 5 good images of one scene with very small differences between them doesn't do anything for me, so even though the other 4 may be good, away they go...

750GB of usable space should last for a VERY long time if you're discriminating about what you keep. With my D2X, a saved TIFF might be around 35MB, give or take. 750GB would allow me to store 20,000 TIFFs or 35,000 RAW files. On a good day in the field, I might get 10 images that are worth keeping, 1 or 2 of which are worth displaying.

There's a lot of good information here. I really agree with Jesse in that the proper workflow and more selective shooting is the first thing you should address (doesn't cost anything to do it). One thing that I've forced myself to do over the years is never simply dump the contents of my memory cards onto my hard drive. Once you do that, it becomes exceedingly difficult to go back and delete unwanted images. I always select the images I'm going to keep while they are still on the card and only copy those onto my hard drive. It has worked wonders for keeping my storage needs to a minimum after years of shooting.

Another issue that hasn't been discussed is the use of sRAW. Selecting sRAW1 (which yields a 10mp image) for images that aren't as important will keep your storage and processing needs down as well. I know I plan to utilize this feature that way.

I have to agree about being more concious of what I'm shooting and what I'm keeping. I've been focusing on this a lot more as well. That was one thing I liked about the 4x5, makes you slow down and think more about what you're shooting. I'm trying to get better at that with the DSLR. I've started trimming down on my photos taken in the field, and deleting more that I know are never going anywhere and those multiples of the same scene/comp.

Good suggestions for sure as compared to just plugging in more discs...

I agree with Jesse, too; no need to keep junk hanging around in the system. Always work harder to make the best pictures possible in the camera.

I also think that deleting images during the early stages of editing is inherently risky; don't do it. You can't expect to always make the best decisions on the first few looks at the images. Wait to delete rejected images until later.

Also, if you don't delete anything during editing you eliminate the risk of making a mistake and accidentally deleting something you shouldn't have.

The efficient workflow moves from one stage to the next as quickly as possible and provides options for recovery at all times. Just mark photos as picks and rejects (with star ratings as necessary) and filter them to only show what you want to see. Come back to look at them with some time gone by; you'll have better perspective in making your final decisions on what to keep or trash permanently.

Shoot carefully, conscientiously and mindfully... but take as many frames as you need to get that perfect shot. There's really no need to hold back; don't let the amount of data/file size be intimating... with digital imaging, more data really is better. And continually upgrading hardware is part of the process.

Imagine having a million of your own photos, taking up 10, 15 or 20 TB... this will be reality for many of us in the not-so-distant future. You'll be much better off if you start getting your systems set up to scale to this as soon as possible.

Don Grall ....I tried Lightroom 1.1 but thought it was too confusing in terms of how it kept track of where everything was. Also when I disconnected the external hard drives, suddenly LR of course had no idea where any of the images were. I turn off the externals when I know I won't be accessing them to prolong their lives. I love LR's interface and large preview window and the fast manner in which you can handle images, but the catalog functions just don't mesh with how I work.

Don, I'd highly recommend trying LR2. It now does a much better job of managing multiple external drives.

Great info here folks. I really appreciate everyone's thoughts.
Thanks,
Paul

This is an excellent discussion on many fronts since I am consdering an upgrade at least to a 50D or maybe the 5dII. Much to think about. My computer is about 5 yrs old and 2nd hd 200 gb drive is just over half full. Any one know if Elements 5 has any file size limitations? Will I need to upgrade to CS3 or 4 when it comes out?

File size limitation with Elements 5 is 300,000 x 300,000 pixels, so it will handle uninterpolated files from either of those cameras..
You can find out how to overcome this limit (and more) in this Adobe tech. note.

If you want to use Adobe Camera Raw with those cameras, you'll need to update your version of Elements to v.7