I've been working with a lot of people on database/catalog issues within Lightroom lately so I wrote an article about it... if you use Lightroom please read the article on my blog and let me know if you find the information useful (or if you have any questions). Thanks!
Good stuff Nat, I found the info useful and informative, since I don't work in LR very often, (because I am not out shooting very often right now) I have to relearn everything, but it is a good reminder to take these precautions when I am editing. Every little bit of info on LR is helpful to me. Thanks for taking time to inform and help.
Good article Nat! Coincidentally, Matt Kloskowski just wrote about this topic in his Lightroom Killer Tips blog. It includes a nice short video.
P.S. While SQLite is a great, lightweight database well suited for Lightroom and similar apps, some in the enterprise software industry might take issue with the characterization of SQLite being very robust. But that doesn't matter too much in the context of this discussion...
I enjoyed your articles, Nat. Thanks for the time involved.
How do you currently manage your catalogs? Do you keep multiple smaller catalogs or one large one?
Have you seen or heard any issues with larger catalogs?
I've got catalogs for each year but I often want to view collections across years so at some point I usually end up importing into a single, large catalog. SQLlite should be able to handle large record sets but smaller databases should mean faster access in general.
Also, do you keep your edits in xmp sidecar files? I've alternated between turning them off to boost performance to keeping them on as insurance if I were to lose a database without a backup (less likely these days).
Kalin -
My master image library is in one catalog file, currently around 32,000 images. I've had no problems with the Lightroom database besides those I caused myself. Julianne Kost said she ran with 100,000 images in a Lightroom database also with no problems. I suspect there are quite a few people with hundreds of thousands in a single catalog.
But I do use lots of Lightroom databases. I use them to move and copy files, to convert to DNG (at various later points in the workflow; sometimes I only have time to copy camera native in the field), and I use temporary working databases in the process of cleaning up my master database.
Export As and Import From Database (under the File menu) gives you lots of safety when it comes to making big changes when cleaning up your catalog. When you Export as Catalog, all your Lightroom edits go into the exported catalog, including Virtual Copies, Collections, Stacks and Previews. If you can benefit from isolating a large group of images to make an improvement to your catalog, consider Export as Catalog first; do your work on whichever database you need to, then Import from Catalog. Everything will transfer; it's a very nice feature.
So I advocate keeping one master catalog for all images in your library plus temporary working databases as needed.
One of Lightroom's real strengths is Collections. To make the most of Collections all your images should be in one catalog. Otherwise you'll be doing lots of additional Export/Import.
All my raw files are DNG. The metadata gets saved right into them. I don't keep the original .CR2 files. So I don't have sidecar files.
I Save Metadata manually, frequently, such as after a round of applying stars. Just like in any other program, the Save command is essential! (But I don't like the the Automatic Save at all.)
Re: faster access to the database - these records are so small and simple in Lightroom (compared to databases in other applications) that I don't think most computers running Lightroom would hiccup when retrieving the results of queries.
What slows Lightroom down is rendering previews. I once had my machine basically "lock up" (but not really) for 30 minutes crunching previews after I selected an upper-level folder. During the building of the preview I couldn't do anything else in Lightroom and I wondered if it would recover. So I just let Lightroom do its thing and all was OK.
Which raises the point: If you can ever avoid it, don't force quit Lightroom. If it looks like the program is locked up - even if it says it's not responding - wait a bit.
The more I use Lightroom the more I love it. It's still a very young program but the future is looking really bright. And what's coming in version 2.0 is truly amazing.
Lots of this stuff (including v2.0) will be covered in detail in my Lightroom book due out this fall.
Ahh... you're writing a Lightroom book! What a surprise.
Your article covers a number of very important issues. Thanks for the summary.
Kalin - like Nat, I also have a single catalog with ~ 38,000 images. Access to any subset/collection is quite fast. So far, no database issues - but I am still wary. Currently all the files are stored on two external 500 GIG hard drives. I also convert my raw files to DNG on import, which eliminates the extra sidecar files for metadata.
Because I frequently edit and process new images away from home, I usually load new shoots into their own LR database. After the initial processing, everything is exported to a catalogue and reimported into the LR master. Simple, robust.
Bill - thanks for describing your methods; my setup sounds very similar ro yours. I use one 500GB drive as my master library, also contains all the Lightroom files. This is synchronized nightly to a twin 500GB drive. A third backup is kept off site and synchronized every few weeks.
When the main library drive fills up, I will move everything to TB drives.
I also do a lot of processing in the field using temporary working catalogs and I agree simple, robust describes managing multiple databases in Lightroom.
And like you, processing in the field, I start with a fresh, empty database for each trip. As I import from CF card, I convert to DNG, add all metadata and keywords, and start rating. I may even make Collections, Virtual Copies, Stacks etc. as necessary. I keep all the shots from the trip in one LR database.
I come home with files ready for final processing and printing. At home I connect my laptop to my main working desktop computer by Ethernet and copy the files from the laptop to the main external drive. In my main catalog, I do an Import from Catalog and everything gets merged into the master database.
Then I delete all the temporary files from both computers.
If I don't take my laptop on the road, I use my SmartDisk FlashTrax 40GB to hold the .CR2 files and I do all the work back at home.
Posted by Kalin Wilson on May 12, 2008 at 11:55 PM
Thanks guys, it's nice to hear about other workflows!
Discussion
8 repliesP.S. While SQLite is a great, lightweight database well suited for Lightroom and similar apps, some in the enterprise software industry might take issue with the characterization of SQLite being very robust.
How do you currently manage your catalogs? Do you keep multiple smaller catalogs or one large one?
Have you seen or heard any issues with larger catalogs?
I've got catalogs for each year but I often want to view collections across years so at some point I usually end up importing into a single, large catalog. SQLlite should be able to handle large record sets but smaller databases should mean faster access in general.
Also, do you keep your edits in xmp sidecar files? I've alternated between turning them off to boost performance to keeping them on as insurance if I were to lose a database without a backup (less likely these days).
Kalin -
My master image library is in one catalog file, currently around 32,000 images. I've had no problems with the Lightroom database besides those I caused myself. Julianne Kost said she ran with 100,000 images in a Lightroom database also with no problems. I suspect there are quite a few people with hundreds of thousands in a single catalog.
But I do use lots of Lightroom databases. I use them to move and copy files, to convert to DNG (at various later points in the workflow; sometimes I only have time to copy camera native in the field), and I use temporary working databases in the process of cleaning up my master database.
Export As and Import From Database (under the File menu) gives you lots of safety when it comes to making big changes when cleaning up your catalog. When you Export as Catalog, all your Lightroom edits go into the exported catalog, including Virtual Copies, Collections, Stacks and Previews. If you can benefit from isolating a large group of images to make an improvement to your catalog, consider Export as Catalog first; do your work on whichever database you need to, then Import from Catalog. Everything will transfer; it's a very nice feature.
So I advocate keeping one master catalog for all images in your library plus temporary working databases as needed.
One of Lightroom's real strengths is Collections. To make the most of Collections all your images should be in one catalog. Otherwise you'll be doing lots of additional Export/Import.
All my raw files are DNG. The metadata gets saved right into them. I don't keep the original .CR2 files. So I don't have sidecar files.
I Save Metadata manually, frequently, such as after a round of applying stars. Just like in any other program, the Save command is essential! (But I don't like the the Automatic Save at all.)
Re: faster access to the database - these records are so small and simple in Lightroom (compared to databases in other applications) that I don't think most computers running Lightroom would hiccup when retrieving the results of queries.
What slows Lightroom down is rendering previews. I once had my machine basically "lock up" (but not really) for 30 minutes crunching previews after I selected an upper-level folder. During the building of the preview I couldn't do anything else in Lightroom and I wondered if it would recover. So I just let Lightroom do its thing and all was OK.
Which raises the point: If you can ever avoid it, don't force quit Lightroom. If it looks like the program is locked up - even if it says it's not responding - wait a bit.
The more I use Lightroom the more I love it. It's still a very young program but the future is looking really bright. And what's coming in version 2.0 is truly amazing.
Lots of this stuff (including v2.0) will be covered in detail in my Lightroom book due out this fall.
Your article covers a number of very important issues. Thanks for the summary.
Kalin - like Nat, I also have a single catalog with ~ 38,000 images. Access to any subset/collection is quite fast. So far, no database issues - but I am still wary. Currently all the files are stored on two external 500 GIG hard drives. I also convert my raw files to DNG on import, which eliminates the extra sidecar files for metadata.
Because I frequently edit and process new images away from home, I usually load new shoots into their own LR database. After the initial processing, everything is exported to a catalogue and reimported into the LR master. Simple, robust.
When the main library drive fills up, I will move everything to TB drives.
I also do a lot of processing in the field using temporary working catalogs and I agree simple, robust describes managing multiple databases in Lightroom.
And like you, processing in the field, I start with a fresh, empty database for each trip. As I import from CF card, I convert to DNG, add all metadata and keywords, and start rating. I may even make Collections, Virtual Copies, Stacks etc. as necessary. I keep all the shots from the trip in one LR database.
I come home with files ready for final processing and printing. At home I connect my laptop to my main working desktop computer by Ethernet and copy the files from the laptop to the main external drive. In my main catalog, I do an Import from Catalog and everything gets merged into the master database.
Then I delete all the temporary files from both computers.
If I don't take my laptop on the road, I use my SmartDisk FlashTrax 40GB to hold the .CR2 files and I do all the work back at home.