On iPhoto to Aperture 3 migration difficulties and new hard drives

Just a few quick notes as a follow up to the article I wrote last week entitled Aperture 3 announced - time to upgrade from iPhoto ‘09?:
1.) migration not so easy
2.) damn, modern hard drives are fast

Explained after the break.

iPhoto '09 to Aperture 3 migration Nawt Sew E-Z, Akshully (at least for me)

OK, so I have a ridiculously large iPhoto library- even after spending a day (more like two) trying to clean out the dupes in my iPhoto collection, it is still over 70,000 images and pocket cam videos. But clean enough I was ready to migrate over to Aperture 3. To prep it, did some reading, and decided to clean up the iPhoto database. Turns out if you hold down option and command as you launch it, you'll get this screen

(publishing difficulties, will update when can)

It includes options on rebuilding thumbnails, recovering orphaned photos, repairing permissions in the library, rebuilding the database from automatic backups, and reclaiming unused space from the database. I did all this (took a while) in preparation.

Reading up on the options, you can have Aperture 3 leave the photos where they are or copy them to a new location. Since my goal was NOT to add Aperture as an option while keeping iPhoto as the primary tool, but instead to migrate entirely over to Aperture 3 as a replacement for iPhoto '09 entirely. But since I'm just running the 30 day demo (19 days left at this point), I wanted to keep my existing library intact.

I did this by choosing a new location for the picture library. Scouting around my Mac Pro I realized I didn't have an extra nearly 800 GB of room handy. Since I wanted to keep this a fully reversible experiment in case I didn't like the results, I got impatient and just drove a few blocks over to the local MacMall outlet in Santa Monica and picked up a new 2TB SATA drive, model Hitachi HDS722020ALA330. While discussing data backups on CML (cinematographer's mailing list), somebody pointed out that the 1.5TB drives I've been installing in my Mac Pro have been unusually problematic, and that hopefully the 2TB models would be better - glad I went for the bigger drive.

Big, Modern New Drives are Superfast

About that new drive - I remember doing RAID tests for uncompressed HD in the early days of HD for Indies, and getting peak drive performance in the 60 MB/sec range, falling off to the 30-35 MB/sec range. For those who don't know or remember this, hard drives vary in speed depending on where the data is written. Imagine a record player and its needle - if you drop the needle at the edge of the record, where the radius is largest (furthest from the center spindle), the needle's speed over the surface of the record is greatest. This is the fastest part of the hard drive - at the edge. As data is written to the disk, it fills up, and new data is written closer and closer to the hub/spindle/center of the disk - where it is slowest. So this new drive? 2TB model, about $220-ish with tax, is waaaaaaaay fast. How fast? When I was writing zeroes to the disk with Disk Utility (for the sole purpose of writing some data to the entire disk and automatically mapping out bad sectors if zeroing doesn't work), I saw it was writing around 130 MB/sec - RAID speeds from a few years ago! In theory, you could record 1080p24 8 bit 4:2:2 video to disk UNCOMPRESSED at this speed. Yowza - that is FAST. By the time it was almost done, write speed had dropped to around 70 MB/sec, still quite speedy. It probably dropped a touch more before finishing, but still - very fast. So a RAID built off of these drives would be quite fast indeed.

Easy Migration with a big library? Fail. At least for me

Next I started an import of my existing iPhoto '09 library into Aperture 3 - there's a File==>Import==>iPhoto Library option. You can tell it whether to leave the images where they are (just creating references to them) or to copy them to the Aperture 3 library - which defaults to your User/Pictures folder, but I didn't have enough room on my boot drive, so I told it to go to the new MacProData2TB drive. I got it started, saw it was going to take a while, so left running while I went out for the night.

Next day - uh oh, got a crash report up on screen, Aperture 3 has unexpectedly hosed itself. Hmm. What now? I open up Aperture 3 and it starts whirring away on my drives. I call up the Activity window (Window==>Show Activity), and it says something about processing 30,000 or so items. Hmm...is this it building previews/sucking out metadata for the remainder of images it hadn't finished, or importing images still, or what? I could see many preview icons for Events, some with picture previews but most blank.

Left it chugging - I can't remember whether it finished (think so) or it might have crashed again.

Next time I sat down to keep trying to push this project forward, I scanned around and realized it had never finished importing all the original items - the import didn't finish. Drat.

Thinking about what to do, I remembered there was a "Do not import duplicates" option in the import options - so I figured I'd just reimport and tell it to skip dupes, and hopefully it would be smart enough to skip over what it had done and not import dupes and then start importing where it had left off.

Fail. As I watched the screen, I could see dated events duplicating themselves - same date in the name, same picture in the preview, same # of photos in the Event. Digging deeper, same named photos, no modifications to the image (which I could see leading to dupes or multiple edited instances of the same image). OK, this wasn't working.

"I say we nuke the entire site from orbit, its the only way to be sure."

When in doubt, nuke and pave. With over 70,000 images in the library, a tweezer job of trying to pluck out the duplicates was not how I wanted to spend my time. So I deleted the library file for Aperture, deleted the folder of imported images on that new drive (all my source images still safe on the original other drive), and thought about what to do next. I relaunched iPhoto with option command held down and selected all the options - I wanted to make this as clean as possible a dataset to go into Aperture 3. That took quite a while.

2nd Migration Attempt Notes

Next day (today), came in and made a new library, made a new folder for pictures to copy into from the import, and did some reading on the support forums. Couple of things I learned:

1.) Having Faces on seems to slow things down considerably, and possibly incurs a memory leak. So I turned it off in the preferences for now - presumably my prior Faces data will still import, after verifying all images successfully in, THEN I'll turn on Faces and see how it goes.

2.) Video files aren't caught by the Don't Import Duplicates function - wonderful. But my earlier duplicates were DEFINITELY photos, I checked - JPG file extension not AVI.

So far it is running OK - looking at Activity Monitor, Aperture's CPU draw is 100-200%, it is running 1500-1700 threads (1500ish now, was 1700ish earlier), RAM usage has gone from an earlier 900MB during import up to 1.25GB as of now, and virtual memory has drifted in the same timeframe from 2.9 to 3.13 GB. I'll see how this turns out - it is currently working on number 300 out of 1764 events to import....

More later,

-mike

Update, later that day - somewhere north of event 650 out of 1764, it crashed again. Reopening it, it starts grinding drives, and I open Activity to see what's up. It is "Updating Library", "Processing" 31,705 items, starting from zero and inching its way upwards, about one item every few seconds. Yep, that slow. So is there a bad item that is making it break?

I'm going to let it finish processing, but I'm about half an inch from f*ck this - I've got better things to do. 3.01, where the *&(*^& are you?

One good note I verified while I had half-a-library imported - Desktops, Screensavers, AND AppleTV can all use Aperture events/pics instead of iPhoto - you get a pop-up menu to let you pick which you want to choose between, iPhoto or Aperture - yay! I really enjoy having a mix of "All time very best, better of last 90 days, and Last Import" on rotation on the 60" living room HDTV while NPR podcasts play on a Sunday.

-mike

further update that night - got an email from a friend who said it makes a cache roughly equal in size to the import.

a.) that could explain it - 750+GB of data being imported, 400GB free on boot drive. Specifically put library file and copy destination on new 2TB drive trying to avoid these issues

b.) if this IS the problem, that is dumb - libraries can be large, and needing a cache size equal to library size, and requiring that it go on boot drive, is brokenly limiting.

UPDATE - SAGA CONTINUES - Apple released 3.0.1 today, but Monday I accidentally deleted my iPhoto Library. Oopsie. Details here on what I did and how one can try to address the situation.