Several months ago I realized that I had about 4 different systems for saving and keeping my photos and this made the recovery from a hard disk crash very difficult. So I have been spending lots of time figuring out how to do it. I have been reading the The DAM Book by Peter Krogh (NPN review). The Antarctic trip really hit my system really hard as I have kept about 5500 images from a little over 8800 captured. SO I stopped filing them until I had a good idead of how I should do it in the future. I have gotten all but about 1000 images up to the most recent standard I had been using with the idea that it will stage a huge migration to my latest and greatest system.
I am now using a number of the things he discusses, in particular he pointed out my one problem in archiving. I was organizing my digital files the same way I have my slides. I really like his bucket system it is very simple and fits the way I think and it seems to bethe best way to archive and back up to me at this point. Mark Ferguson and I talked about this for quite a while one evening.
I had never really thought how important this issue could be. In December/January it took me a huge amount of time to put together a group of images that were requested by a photoeditor I have worked with, considering the time I invested in getting these images together I got paid a very small amount of money for my time.
Enough rambling, now to my question. Peter is a big proponent of using DNG to archive your raw files. How many of you out there use DNG as an archive? Are there any issues using it? I think Nat is using it if I remember correctly.
Scott, I'm glad you brought up this topic. I'll tell you a little about my workflow and then add a few questions for discussion. To start, I keep a very small percentage of the pictures that I take (compared to your 5 out of every 8). After I download my pictures to my computer, I view the raw files in Adobe Bridge and decide which ones are even worth the time to edit and throw the rest away. Next, I give each Raw file a title that includes the Data, Title, and Location. I edit each picture as needed in Photoshop and save an EDIT copy and a FINAL copy. The raw file and the two PSD files have the same file name with the exception of the file extention (in my case: .CR2) and the EDIT and FINAL in the two PSD files. I dump all raw files in my RAW file folder. Next, I organize the PSD files by Location and Date. I've never bothered with DNG because I'm not sure how long it will be around. It's not exactly widely used yet. I'd like to add the following questions to this discussion:
1. What is the best way to organize files (By Date, Location, Genre, etc??) for later use.
2. What is the advantage of DNG when compared to Raw and PSD?
Good topic for discussion, and I'm sure you'll receive plenty of differing ideas.
As for DNG, the major manufacturers are simply not embracing it - therefore, I don't believe it will be long lasting.
So, what format should you use for archiving? Tim Grey (www.timgrey.com) recently addressed this same question in a recent DDQ. His opinion was that since JPEG format is so prolific, that is a good one to consider and the format should last a long time. For the better stuff, either RAW or TIFF should be used; with TIFF probably the longest-lasting.
For me, I primarily shoot high-res JPEGS for the routine pictures and RAW for the more serious pictures (commercial, weddings, fine art, etc.). This optimizes the file storage issue. I plan to keep a close watch on whether Canon or Nikon considers changing their RAW format. If they do, I might use a batch process to start converting my RAWs to TIFFs (and buy more hard drives in the process). Of course, performing major edits of your images would reduce the archival problem, but that's a discussion for another date!
As far as organizing your photos, that's probably a separate discussion, as well! Mine are organized by subject - then sub-topics. For example, "Nature" - then under that, "Fall Color", "Sunrise/Sunset", "Spring", "Water", "Wildlife", etc. Then, under those sub-topics would be specific folders of different shoots with place/date in the folder name. The images themselves get renamed to the place and date with in the filename.
As far as organizing/editing software, I've tried about all there is. Most are too slow when you have several hundred to edit. Some take forever to rename files. Some are clumsy to work with. I originally used "PIE" (Picture Information Extractor) - later becoming, PIE Studio. That was really fast, but didn't do RAW files. I also used ACDSee (still do a little). It's a good all-around sorting/editing/sharing software, but gets bogged down severely when dealing with a few hundred images.I've tried using Photoshop (up to CS) and it was always excruciatingly slow. I haven't used Bridge much, because (for all versions of PS), I don't care for the way Adobe uses "boxcar" files for the EXIF and IPTC info. It's just a whole extra level of file management I'd rather not have to think about. Then, what if these files get separated from the image file? Yuck!
Other organizers I've tried, but don't really care for include Adobe Photoshop Album, Bibble or Apple's iPhoto.
Lately, I've been using Photo Mechanic. It's still not the perfect software, but it's gradually improving. It's fast at sorting and is not bothered by hundreds of files. Creating IPTC data is fast and different IPTC templates may be saved. This data can be added (and files renamed) during the "injest" process. Minor editing and sharing can also be done within the program. I also create key words that are saved within the IPTC data. This allows IPTC-compatible programs to find specific images during searches.
I'm currently evaluating the beta version of Apple's LightTable. I also hear good things from BreezeBrowser (Breeze Systems).
I agree with Tim Grey's opinion and take a wait-and-see approach on DNG. If it does become more widely accepted, I'll re-evaluate.
For archiving I store RAW files, edits in psd, and finals as TIFF. I don't expect any particular file format to last forever, but I assume there will be plenty of warning when a format falls out of favor.
For organization, I prefer a geographic approach for landscape/nature files:
State (Colorado)
----- Area (San Juans)
---------subTopic (Blue_Lakes_Trail)
-------------date-subject (20050409_LowerBlueLake)
-------------------RAW, psd, TIFF, JPEG files
-------------date-subject (20060507_UpperBlueLake)
-------------------RAW, psd, TIFF, JPEG files
-------------etc.
I use Adobe Bridge to manage these folders.
2 copies are stored on separate hard drives.
RAW files are written to DVD ASAP after acquisition
As long as we're discussing archiving...one thing I failed to mention in my post above was backing up your files. Anyone have to recover their data following a hard drive crash? HDD crashes are just a matter of time - right? Also, I've heard plenty of horror stories regarding unreadable CD-R, CD-RW and DVDs. Not to mention the fact there are so many optical formats - how will you know whether the CD-R you saved data to will not be obsolete in n-number of years? They're also sensitive to sunlight and are easily scratched.
Personally, I have been using a multiple (3) HDD backup system. I place the purchase date on each one and replace the oldest each year. One of these drives is stored offsite. Backups occur weekly (or more often if I save a large, or important, job). So far, I haven't been disappointed...
Posted by Scott Buckel on July 9, 2006 at 10:09 PM
Ken - I have been archiving my raw and proces files on DVD. I had a HDD crash on Christmas Eve 2005. I spent the next week recovering my financial data as well as my digital and scanned files. I found that I had to use all three sets of DVDs to recover my files since no one set was free of corrupted files. I managed to recover all but 8 files. This prompted me to buy more storage. Now I have data on three different HDD much the same way you are doing. My backs are done manually about one a week and I plan on setting up an auotmated way as soon as I complete the migration.
As soon as I migrate to the archive system promoted by The DAM Book, I will have a better system to recover from a crash. In short this system archive all of your raw files in buckets the same size as your archive media. For DVD I will have folders with about 4.3 gb and the folder name will be DVD_NNN. I will be using meta data to find any image that I want to either print or submit for outside use. I will be moving to iView Media Pro from Digital Pro.
I plan to write up the system I am using for the tips section of the website I have been working on forever.
Scott
Posted by Scott Borger on July 9, 2006 at 11:34 PM
Use a hard drive backup AND archival DVD's- Matsui brand. Matsui gold is supposed to last the longest. Store DVD's in a dark, dry place, away from extreme temps. NEVER write on the actual disc with a Sharpie,etc. The acid will eat through the Data portion of a disc. Yes, hard drives will eventually fail after an unknown number of rotations. You can't have too many backups. I suggest keeping one of them off location in case of flood, fire, or theft.
Good topic. I have been playing with the DNG format, but only because PS CS doesn't support my Digital Rebel XT. So I use Adobe's latest DNG converter to take the raw files to the DNG format which PS CS can then process just like a normal raw file from the camera. Works great, but does create double the amount of files. When I upgrade PS to a version that can read the XT raw files directly, I'll have to decide if I should just dump all the DNG's since I will no longer need them.
I have been using a date hierarchy for my storage method, but I've wanted to use iMatch for a while now; just haven't gotten around to doing it. But with two young children and landscapes, I'll have to find something soon to search for images based on metadata.
Backups; never ever forget your backups. I use the triple hard drive method too. Nightly incrementals to another internal drive and a monthly full to yet another offsite drive. If a do a lot of shooting I do the offsites more often. Works much faster than DVD's and at a lower cost if you use good (Matsui Gold) media.
Craig
Posted by Arthur Lund on July 11, 2006 at 10:20 PM
The 1st thing I do is backup my cards to Mitsui CD-Rom with a stand alone cd burner
that I take with me in my car. It runs on batteries, a cigarette converter, or A.C. I use CD-Rom because of considerable articles that point out that the DVD format standard is still
up for grabs. !Scott brought up the "DAM" book. I recommended it to everyone of you
before, and I heartily recommend it again. Anybody who values quick access to their
photos need to buy it and read it!!!!
Discussion
8 repliesI am now using a number of the things he discusses, in particular he pointed out my one problem in archiving. I was organizing my digital files the same way I have my slides. I really like his bucket system it is very simple and fits the way I think and it seems to bethe best way to archive and back up to me at this point. Mark Ferguson and I talked about this for quite a while one evening.
I had never really thought how important this issue could be. In December/January it took me a huge amount of time to put together a group of images that were requested by a photoeditor I have worked with, considering the time I invested in getting these images together I got paid a very small amount of money for my time.
Enough rambling, now to my question. Peter is a big proponent of using DNG to archive your raw files. How many of you out there use DNG as an archive? Are there any issues using it? I think Nat is using it if I remember correctly.
Thanks,
Scott
1. What is the best way to organize files (By Date, Location, Genre, etc??) for later use.
2. What is the advantage of DNG when compared to Raw and PSD?
As for DNG, the major manufacturers are simply not embracing it - therefore, I don't believe it will be long lasting.
So, what format should you use for archiving? Tim Grey (www.timgrey.com) recently addressed this same question in a recent DDQ. His opinion was that since JPEG format is so prolific, that is a good one to consider and the format should last a long time. For the better stuff, either RAW or TIFF should be used; with TIFF probably the longest-lasting.
For me, I primarily shoot high-res JPEGS for the routine pictures and RAW for the more serious pictures (commercial, weddings, fine art, etc.). This optimizes the file storage issue. I plan to keep a close watch on whether Canon or Nikon considers changing their RAW format. If they do, I might use a batch process to start converting my RAWs to TIFFs (and buy more hard drives in the process). Of course, performing major edits of your images would reduce the archival problem, but that's a discussion for another date!
As far as organizing your photos, that's probably a separate discussion, as well! Mine are organized by subject - then sub-topics. For example, "Nature" - then under that, "Fall Color", "Sunrise/Sunset", "Spring", "Water", "Wildlife", etc. Then, under those sub-topics would be specific folders of different shoots with place/date in the folder name. The images themselves get renamed to the place and date with in the filename.
As far as organizing/editing software, I've tried about all there is. Most are too slow when you have several hundred to edit. Some take forever to rename files. Some are clumsy to work with. I originally used "PIE" (Picture Information Extractor) - later becoming, PIE Studio. That was really fast, but didn't do RAW files. I also used ACDSee (still do a little). It's a good all-around sorting/editing/sharing software, but gets bogged down severely when dealing with a few hundred images.I've tried using Photoshop (up to CS) and it was always excruciatingly slow. I haven't used Bridge much, because (for all versions of PS), I don't care for the way Adobe uses "boxcar" files for the EXIF and IPTC info. It's just a whole extra level of file management I'd rather not have to think about. Then, what if these files get separated from the image file? Yuck!
Other organizers I've tried, but don't really care for include Adobe Photoshop Album, Bibble or Apple's iPhoto.
Lately, I've been using Photo Mechanic. It's still not the perfect software, but it's gradually improving. It's fast at sorting and is not bothered by hundreds of files. Creating IPTC data is fast and different IPTC templates may be saved. This data can be added (and files renamed) during the "injest" process. Minor editing and sharing can also be done within the program. I also create key words that are saved within the IPTC data. This allows IPTC-compatible programs to find specific images during searches.
I'm currently evaluating the beta version of Apple's LightTable. I also hear good things from BreezeBrowser (Breeze Systems).
Ken
For archiving I store RAW files, edits in psd, and finals as TIFF. I don't expect any particular file format to last forever, but I assume there will be plenty of warning when a format falls out of favor.
For organization, I prefer a geographic approach for landscape/nature files:
State (Colorado)
----- Area (San Juans)
---------subTopic (Blue_Lakes_Trail)
-------------date-subject (20050409_LowerBlueLake)
-------------------RAW, psd, TIFF, JPEG files
-------------date-subject (20060507_UpperBlueLake)
-------------------RAW, psd, TIFF, JPEG files
-------------etc.
I use Adobe Bridge to manage these folders.
2 copies are stored on separate hard drives.
RAW files are written to DVD ASAP after acquisition
Personally, I have been using a multiple (3) HDD backup system. I place the purchase date on each one and replace the oldest each year. One of these drives is stored offsite. Backups occur weekly (or more often if I save a large, or important, job). So far, I haven't been disappointed...
As soon as I migrate to the archive system promoted by The DAM Book, I will have a better system to recover from a crash. In short this system archive all of your raw files in buckets the same size as your archive media. For DVD I will have folders with about 4.3 gb and the folder name will be DVD_NNN. I will be using meta data to find any image that I want to either print or submit for outside use. I will be moving to iView Media Pro from Digital Pro.
I plan to write up the system I am using for the tips section of the website I have been working on forever.
Scott
I have been using a date hierarchy for my storage method, but I've wanted to use iMatch for a while now; just haven't gotten around to doing it. But with two young children and landscapes, I'll have to find something soon to search for images based on metadata.
Backups; never ever forget your backups. I use the triple hard drive method too. Nightly incrementals to another internal drive and a monthly full to yet another offsite drive. If a do a lot of shooting I do the offsites more often. Works much faster than DVD's and at a lower cost if you use good (Matsui Gold) media.
Craig
that I take with me in my car. It runs on batteries, a cigarette converter, or A.C. I use CD-Rom because of considerable articles that point out that the DVD format standard is still
up for grabs. !Scott brought up the "DAM" book. I recommended it to everyone of you
before, and I heartily recommend it again. Anybody who values quick access to their
photos need to buy it and read it!!!!
Art Lund