Advanced keying with After Effects: Part 1

Chris Zwar

6 years ago

Advanced keying with After Effects: Part 1 4

Nothing says visual fx like a greenscreen. The image of a film set covered with blue or green fabric has become synonymous with visual effects. Keying has become such an important part of video production that it’s easy to underestimate how much there is to learn about the process, and it’s easy to overestimate how easy it is.

In this 5-part series, I’ll be looking at chromakey in After Effects and offering a range of background information, tips, tricks, trivia, and general insight that’s come from working with After Effects for over 20 years.

This series has been a long time coming, and I’ve been intending to write something about keying for over ten years! When I started working as a freelancer, I set up a basic website. It was always neglected but occasionally I’d write a blog post so there was more than just a showreel. At some point, I scribbled down some thoughts on keying – a fairly unstructured account of how keying had changed over the previous ten years. As a result of that blog, I was invited to submit articles for the ProVideo Coalition and I’ve been posting the odd article here ever since. So every time I write something for the PVC, I think back to how my thoughts on chromakey started it all.

In 2014 I started writing an article on keying and ended up with one on how TVC production has changed since 1997…

Ever since then I’ve intended to put together a tutorial on keying, and some of the notes I jotted down over the years have gone on to become stand-alone articles in their own right. So far, the largest and most ambitious article I’ve made for the ProVideo Coalition was a 50-minute series on the desktop video revolution.

But believe it or not, that series actually started out as the introduction to this series on keying! There are several recurring themes in my other articles that originally began as notes for a series on chromakey, and they can all be traced back to my original post “the myth of the single click”.

The myth of the single click is a term I came up with to talk about the most important aspect of keying. It’s something that is the basis for this entire tutorial series. If you’re wondering what I think the single most important thing is when it comes to keying, then it’s not something technical. It’s not to do with the camera or lighting. It’s not a particular plugin or other piece of software. It’s not a secret technique hidden away in After Effects that only 3 people know about. It’s nothing like that.

It’s attitude.

You might think that’s a weird thing to say – what does attitude have to do with keying?

The myth of the single click: Origin story

In my case the answer goes back to 1997. 1997 is a year I mention in many of my articles, as it was significant for two reasons. Firstly, it was the year I began working professionally. Secondly, it was the year that Media 100 and Avid released desktop video products capable of what we called online quality video. Before then, desktop video was an offline tool. You didn’t finish or output final videos using your desktop computer. It’s something we take for granted now, especially as phones can record 4K video, but way back in the 1990s the idea of desktop video was something new. 1997 was the first year in which you could produce a finished, final-quality video on a desktop computer. This included After Effects version 3.1, which is where I first encountered chromakey.

As I said above, some of the notes I made that were originally intended to be about chromakey ended up becoming separate articles. Explaining how video production has changed since 1997 is one of those cases. The desktop video revolution series demonstrates the difference between a TVC made in 1997, and one made in 2014. The point of the series is that in the late 1990s, high end post production was an exclusive club. You needed very expensive, high-end tools like a Quantel Henry or Discrete Logic Flame that ran on a Silicon Graphics supercomputer.

A range of Quantel products from the 1990s.

I often refer to these as “million dollar rooms”, because a single suite could easily cost that much to install. In the comments to one of those articles a reader noted that their company had actually spent over 3 million dollars setting theirs up. I’m not exactly sure, but I’ve estimated that in 1997 there were less than 10 online suites in all of Melbourne, where I was working at the time.

That’s how high-end visual fx were done in the 90s, there was no other option.

The sheer cost and exclusivity of these high-end tools gave them an aura and mystique, and it was common for desktop editors like myself to assume they were more powerful than they really were. There was no easy way to learn about these machines or how to use them. There was no immediately obvious resource to learn about digital compositing. There was no Video CoPilot, no Creative Cow. It wasn’t even common for a computer to be connected to the internet. Its not like After Effects was a common program; people weren’t doing courses to learn how to key in After Effects 3.1, because no-one had really done keying in After Effects before then.

Ads for keying software like Primatte always seemed to include blonde girls with wispy hair, or smoke. But they never featured blonde girls smoking, so I think the marketing department missed a trick.

At the same time there were ads and magazines that suggested visual fx was easy if you bought the right product. Keying software such as Ultimatte and Primatte appeared in full-page ads in Cinefex, reinforcing the impression that million-dollar machines could magically key just by pressing a button.

During the 1990s, Quantel published its own quarterly magazine called Clips, which showcased work done on Quantel machines from all around the world. One edition included a two-page article on the new chromakey plugin for the Quantel Henry. To someone young, who’d never worked on a system like that, the underlying message was that perfect keying should be easy.

When I sat in front of my 120mhz PowerMac, with After Effects version 3.1, trying to key standard definition interlaced footage captured on an analogue video format and digitised to a compressed codec, it’s fair to say I had something of an inferiority complex. Keying interlaced Betacam SP footage in After Effects 3.1 wasn’t easy, but those fancy ads in Cinefex made me think that the select few who got to work with Henrys and Flames would be producing much better work, because their tools were more powerful.

As desktop technology improved, more keying products came onto the market and more ads appeared. Articles were written, advice was offered in forums and tutorials were made and shared. There was advice on lighting, greenscreen fabrics, cameras, lenses, how to shoot for chromakey, on-set monitoring, what plugins to use – everything. And all of this advice, over an extended period of time, continually reinforced this notion that keying should be easy, and if everything was done properly then you could key with a single mouse click.

Keying has an ugly step-sister: rotoscoping. For the first ten years of my career, rotoscoping was a dirty word. It was synonymous with dreary, repetitive, slow, boring work that was given to juniors. And in the same way that there was always an implication that keying should be possible with a single click, there was the implication that rotoscoping only happened when something went wrong. No one wanted to rotoscope, and all of those amazing ads for perfect keying software were subliminally screaming out that you shouldn’t have to.

there was the implication that rotoscoping only happened when something went wrong

This attitude wasn’t just me. The idea that rotoscoping only happened if something went wrong was just a symptom of the time. The entire industry was coming to grips with the transition to desktop, digital video. A broader range of the video industry was now adopting tools and techniques that had previously only been available to a select few, and so everyone was learning as they went along. This was roughly the same time that video production was possible by anyone with a cheap DV camera and an iMac. Producers were just as likely to assume keying was supposed to be easy as anyone else.

The myth of the single click was not something that was ever really said out loud. No one explicitly said that you should be able to key with a single click, it was simply implied through countless ads, articles, forums, tutorials and so on. And yet it’s just that – a myth.

Gaining experience, not plugins

After several years of working as a freelancer with different companies, one day I found myself working on a project with an entirely digital background. The whole thing had been shot against greenscreen. Much to my surprise, the producer expected that every single shot would require some type of clean up in addition to keying. There was time in the budget to work with the footage and do what was necessary to get the keys looking great. Rotoscoping was expected.

It was the first time I’d worked with a Producer who was familiar with keying, and who treated cleanup and rotoscoping as part of the process.

And finally, I realized that this is how it should be.

With many more years of experience behind me, I still think that’s the most important lesson when it comes to keying. It doesn’t matter what software or hardware you’re using, the first step is to acknowledge that you’re unlikely to get a perfect composite just by pressing a button.

I’m not going to say that keying with a single click is impossible. Of course there will be time when you have footage that gives you a good key very quickly. In some sense I’m not talking about keying with a single click being impossible, but more the overall compositing process.

One of the most significant things I’ve learned is that to get a perfect key, there’s always some degree of rotoscoping required. Even if it’s a simple garbage matte that only takes a few keyframes, there is nearly always some amount of manual masking. It’s difficult to key feet perfectly, so even a simple presenter-to-camera type piece will probably need additional attention on the feet. But reflections and other spill – something as simple as the talent’s watch reflecting the screen colour on random frames – can require manual touch up.

Rotoscoping may never be considered as cool as colour grading but it’s very much an artform that requires skill and experience. It isn’t an easy job that can be given to juniors. It certainly can be boring and repetitive, but this is really just an indication of the level of precision and care that needs to be taken.

Rotoscoping does not mean something has gone wrong. It does not mean someone is to blame. It’s just part of the process and it should be budgeted for accordingly.