Two things:

1.) Apple today released Aperture 3.0.1, addressing many of the issues I raised with version 3.0

2.) I’ve been busy on my iPhoto ’09 to Aperture 3.0 upgrade saga, but did something incredibly dumb, losing me hundreds of hours of effort.

Details on both after the jump.

Executive brief: Apple today released Aperture 3.0.1 fixing many known issues (no guarantees/reports it fixes’em all), and I was stupid and deleted a crucial file, aka the critical and massive iPhoto Library I’d been working on. Remember, kids – your time is more valuable than the cost of yet another hard drive, and the 5 minutes it takes to set up a backup copy rolling before leaving for the night. Even for personal projects. Shut up. It was late. I was on the phone. I was busy. I wish I’d been drinking.

Usual longform Mikey Version follows:

Aperture 3.0.1

Well, lookie here:

“This update improves overall stability and addresses a number of issues in Aperture 3, including:

This update is recommended for all users of Aperture 3.”

Further details available here in the Aperture 3.0.1: Release notes

How Not To Do Things

So I’ve been poking away at consolidating my iPhoto library – about a dozen different iPhoto Libraries scattered across half a dozen different Macs and at least twice that many drives floating around. Once in a single iPhoto Library, it was over 70,000 still images and personal camera videos, a little under 800GB, going back about 12 years. Quite the effort.

I’d managed to get them into One Library To Rule Them All. RAW support had recently come out for my Nikon D300S. I was still busy purging duplicates from the over 70,000 images going back over 12 years. In the process of making room and organizing files and moving gobs of files around to clear room for the One True Archive…I deleted The One True Archive. I was in the middle of renaming some other smaller libraries (I’d recently made some books, and since they don’t migrate over, was making mini-libraries that were just what was needed for those books). I’d duped a library, renamed it, trimmed it down to the stuff for the book. Somewhere in the renaming, deleting of unneeded, etc., I noticed emptying the trash was taking waaaaaaaay longer than necessary. I started digging around and realized I had about 800GB more room than I should have on the drive in question. Then the Bad Tingling began.

My reaction is best exemplified by this expert’s rendition:

This is one of those situations that I considered not posting about, but here’s the deal – I started this blog, in part, to share what I’ve learned, and if you can’t help prevent massive, gouting artery injuries for your readers, what good are you doing anybody? The rest of the deal – I’ve been a Mac user since they came out, over 25 years ago. I’ve made my living with Macs for 20+ of those years, in digital media creation, creating I don’t want to know how many hundreds of thousands of files in that time. I have never, ever, Ever, EVER deleted a crucial file before. And then Emptied the Trash. Yikes. Yes, I did that. And you can look at this as Mike Is Getting Old and Feeble (or even febrile), or If You Wait Long Enough, Everything Happens Eventually.

Backups, you say? With the file residing on a non-boot drive, and too big to fit on my Time Capsule, I was waiting to finish getting it organized as an immaculate iPhoto archive up to date before backing that up and then converting over to Aperture 3.x, so I’d have the option of going back to iPhoto if I didn’t like Aperture. Ahhhhh, rich irony.

So I immediately big the Big Red Panic button and sent out emails to the nerd herd that knows even more than I do. Turns out I was on the right path – I already had one thing recommended, Data Rescue II v1.2.1, and got that rolling. I also bought and downloaded File Salvage which another friend, who has worked at Apple as a developer for years recommended. Data Rescue ran overnight scanning the empty space on the drive in question (SO lucky it wasn’t a boot drive, and I did nothing else to write to that drive after Moment Of Doom).

Things I already knew:
-don’t to ANYTHING that’ll write files to that drive! It’d be overwriting those deleted files I want back
-use PASSIVE methods! Anything that tries to restore in place may only muck things up worse, and ruin chances of deeper/harder/longer recovery efforts

I also have about 40% of the iPhoto library that had successfully imported into Aperture before it kept crashing – so I’m debating building my foundation on Aperture now, since it will have maintained metadata, color corrections, etc. on all those images that did make it over. Thank HEAVENS I didn’t delete it when it didn’t finish! That a all my RAW photos from before RAW support came out for iPhoto on each of my two DSLRs has helped me to keep copies of stuff around – I’m pretty good, apparently, at making sure there’s redundant (even if overlapping) copies of stuff sitting around on another physically separate drive.

What COULD I have done?
-have an external, bigger drive hooked up to my Time Capsule and back up to that (but even then there can be problems – my Apple dev friend passed on a horror story about a Time Capsule recovery to fix a minor problem in iPhoto created vastly larger, destructive ones
-have copied off the collection at an earlier time (requiring buying another drive). I figured “I’ll be done with this in a day or two, what are the odds of something going wrong in that timeframe?” Bwaahahahahhaaaaaaaaa…….crap.

Even if I’d had a fault tolerant RAID, it wouldn’t have protected me from my stupidity on this one – the change would have been instant.

Since I do so much video work and creation of many, many small files (frame sequences and the like), I’ve forgone things like Tech Tool Deluxe that track directory changes, deleted files, etc. – I may change that behavior in the future!

In the end, I think I’ll get 95-99% of the pictures back, but it’ll probably take a week or more of poking at it. What is my time worth again? How cheap are drives? Even for personal work? Oh yeah. That. Do it. Do It NOW.

So in the end, thinking/writing out loud here, my process may end up being:
-rely on the incomplete Aperture 3 library I already have as a base – presuming nothing got broken in the process, metadata, color correction, corrected & original linking are all still intact there
-use iPhoto Library Manager to, one at a time, go through all those copies of old libraries to piece things together, and make one clean library I can import (with Don’t Copy Duplicates turned on) into Aperture 3.x
-each wave of files, append the metadata so when confronted with multiple copies (inevitably) I’ll know what sources are more trustworthy, in this order: OriginalApertureImportSession, RebuiltiPhoto, SourceRAW, RecoveredFiles (those last have no meaningful filename, either – the name is a pixel dimension and a unique number).

Anybody have any better ideas? Other than calling me an idiot for a.) doing this and b.) not backing up beforehand, or at least often enough. File this one under See This Scar Here? That’s Why I Don’t Do That Anymore.

This one still burns – taunting commenters will get the banhammer.

BUT…
-everything I have to pull from recovery will have a new, random name (but at least EXIF info is intact)
-probably, cumulatively, hundreds of hours of tweaky color correction will have been lost
-keywording, metadata, etc. will have been lost
-I have to sift EVERYTHING….again

At least…..
-this time I think I’m doing a better job on the sift, with better tools, and have a better process this time (iPhoto Library Manager, for instance, is a BIG help)
-I’m being more thorough since I have to start almost from scratch, and am finding more pics going further back
-I have some time to do this right now
-I may put the New Stuff on an external (copy of it!) so that I can work on laptop, sift, and watch TV in the evenings. Or somesuch.

This is also why no PMA coverage yet – I went, it was fun, saw cool stuff, but gotta get this breathing again before can stop to write up that stuff.

Kicking myself,

-mike

PS – since I’d been running the demo for precisely the reason why they have a demo – to see if it worked for me prior to purchase – looks like I’ll DEFINITELY be needing to buy it now.

Also, the 3.0.1 update doesn’t show up in Software Update if you’re running the demo, nor will downloading and running the manual updater update it if you are running in demo mode – so you need to download the full demo installer again – I used another email address and the download link now downloads 3.0.1. So AH – maybe I don’t have to purchase just yet….

-mike

NOTES ON DATA RECOVERY

In this case, I had deleted the iPhoto Library (which is really just a package, which is really just a Special Magical Folder – just a folder with a special flag that says “treat this as a double-clickable icon/application,” when in fact it is a folder with a bunch of stuff inside. Applications are packages too – right click and Show Package Contents if you want.

So essentially, I deleted a really big folder and then Emptied Trash. Ow, Ow, OWWWWWCH. Dumb, bad, etc. OK, moving on. At least I figured it out pretty quickly. First thing I did, about 2 minutes after it must have happened, was shut down – I didn’t want anything going on with that drive until I figured out what to do next. Fired up the laptop (this disaster took place on the 8 core Mac Pro), and emailed folks as described above. When I got Data Rescue II v1.2.1 running (came with CF cards ironically), it asked what to scan and how to do so – I elected to do a fine toothed scan of all of the empty space on the drive. This scan took a looooooong time – many many hours – I think I left it running overnight, and was VERY careful to specify ANOTHER drive for it to put its temporary files and recovered files. The next day I filtered through the results and told it to grab all image and movie files and whatever else it could find and left it running. The good news – it found LOTS, and fortunately it scanned for .NEF (Nikon Electronic Format, aka Nikon RAW images from my D90 and D300S) and found tons of those too.

The down side – it found stuff I didn’t care about too, like all the little preview icons for faces, etc. Worse yet, the original file names are all gone – now they are all things like J4304X2852 – 18224.jpg. At least I know the size and file format, but all the original naming, and original creation and modification dates are all now reset to when the files were recovered. At least they have EXIF information that includes date/time originally captured, or I’d NEVER be able to sift out the dupes. As is, I’m going to import all these imported images as suspect, and flag them as such, until proven otherwise, and hopefully only have to rely on these recovered files in and only if all other avenues have failed to produce the originals. But there’s 772 GB of recovered files, I don’t know if any of them are bad/corrupt, and it is….UGH – 235,228 individual files – groan. I’ll filter out by size to get rid of all the previews (besides being in the name, the Finder has ways to sift images based on width and height and other factors – so I can make a smart search to filter out so I can kill off all JPEGs below a certain size, for instance).

OK, back to it….