CMG Hidden Gems: Chapter 12 – Stencils and the “T”

As you are aware by now, we’re going through our book Creating Motion Graphics with After Effects 5th Edition (“CMG5” for short) and pulling out a few “hidden gems” from each chapter. These will include both essential tips for new users, as well as timesaving gems that experienced users may not be aware existed.

Chapter 12 focuses on Stencils, which are relatives of Track Mattes (covered in the previous chapter). Whereas Track Mattes use either the alpha channel or overall luminance of the layer on top to cut out the layer immediately underneath, Stencils use this same information from the layer on top to cut out all of the layers underneath inside a composition. (Technically, since layers render from the bottom up, the stencil is really only cutting out one image – the composite image of all layers below.)

Rather than appearing in the dedicated “TrkMat” mode popup, Stencils appear at the bottom of the Blending Modes popup (as Modes also affect the composite of all layers underneath). Inverted Stencils are referred to as Silhouettes (as in, Silhouette Luma rather than Stencil Luma Inverted). The otherwise short Chapter 12 also covers two often-mysterious functions in After Effects: the Transparency (“T”) switch, and Alpha Add mode.

You Can’t Have Your Cake and Eat It Too

We’ll start with a tip that becomes obvious all too quickly when using Stencils: You can’t combine a stencil and a full-frame background layer in the same composition. Stencils cut out everything underneath them in the layer stack. This is great for filling an alpha channel of, say, text or a client’s logo with a composite of multiple layers, but if you then want to place that result over a full-frame background, the background will get cut out too! Instead, you will need to perform the stenciling in one composition, and then nest that composite over the desired background in a second composition. Once nested, you can also add edge effects such as a drop shadow or glow.

An inverted stencil – Silhouette Luma – is used to create a cutout from a composite of two layers (top left). This composite is then placed in a second composition on top of the desired full-frame footage and a drop shadow is added (top right). Footage courtesy Artbeats Desktop Technology, Digital Moods, and Business World.

Preserve Underlying Transparency

Preserve Transparency – the mysterious “T” switch in the Modes column of the Timeline panel – can be thought of as a reverse stencil. Rather than saying “this layer affects everyone underneath” (which a stencil does), it says “this layer is affected by everyone underneath” – the combined transparencies for all layers underneath affect the transparency of the layer you have turned it on for.

Below left is the combination of three layers created in Adobe Illustrator. The content of the layers is opaque; the area surrounding them is transparent (noted by the Transparency Grid visible around them). Below center is a full-frame piece of footage that is placed on top of those three layers with interesting alphas. Below right is the result of setting the “T” switch for the full-frame layer on top. Underneath the three images is the layer stack in the Timeline panel; note the “T” column between Mode and TrkMat:

Footage courtesy Artbeats/Digidelic.

Why not use track mattes or stencils in this case? Because stencils and mattes borrow transparency from just one layer; here we are relying on the composite of multiple layers to create the final pattern of transparency we wish to preserve. With the Preserve Transparency switch, you would need to nest the text layers off into their own composition, and then use the resulting combined transparency as a matte or stencil in a second composition.

This technique also comes in handy for visual effects work such as keying and rotoscoping, where you may want to combine multiple keys or matte passes to create a composite transparency for a tricky source layer you need to cut out. Concentrate on each element of the source (such as hair, arms, body, etc.) separately in its own layer, then place the original footage on top with its Preserve Transparency switch set to receive the composite of each of these component transparencies.

Add Alpha Mode

While we’re on the subject of unusual transparency tricks, the almost-as-mysterious Alpha Add mode works great with stencils.

For the image of the heart at right, the left side is a nested composition that uses Stencil Luma, and the right side is a second nested comp with the same heart image but using Silhouette Luma (exactly like the left side, but inverted).

These two halves are then combined in a third comp. Where the alpha channels meet along the seam, both antialiased edges have identical transparency values. But instead of adding these values together, the transparency of both layer edges are honored and factored together (50% opaque + 50% opaque = 75% opaque, not 100%). Since this fails to result in a fully opaque pixel, you get a seam. Note that this is not a problem with just stencils – you will see similar problems with track mattes and masking where identical edges meet.

To fix this, select the top layer, and select the Alpha Add mode from near the bottom of the Modes column, as show above. The complementary alpha channels are now added together – as opposed to compositing on top of each other – and the seam disappears. The resulting composite is shown at right.

Bonus Gem: Effecting Stencils

This is a subject not covered in CMG5, but is worth discussing here (especially as this is otherwise a short chapter, without a lot of other gems to uncover).

It’s not obvious how to predictably add edge treatments such as bevels and drop shadows to track matte pairs; the three solutions are to use Adjustment Layers, nesting the track matte pair into its own composition and effecting the resulting layer in a new comp, or the little-known trick of applying a Photoshop-like Layer Style to the layer with the Track Matte mode applied.

The first two also work with stencils and silhouettes; unfortunately, Layer Styles don’t – they break the stencil function:

A typical application of stencils would be having this ampersand illustration cutting out a composite of layers underneath (above left). However, if a Layer > Layer Style such as Bevel & Emboss is applied to the ampersand, the stencil is disabled, the black ampersand gets the bevel, and the layers underneath are no longer cut out (above right). The broken layer stack is shown below.

Trish and Chris Meyer share seventeen-plus years of real-world film and video production experience inside their now-classic book Creating Motion Graphics with After Effects (CMG).

The 5th edition has been thoroughly revised to reflect the new features introduced in both After Effects CS4 and CS5. New chapters cover the new Roto Brush feature, as well as mocha and mocha shape. The 3D section has been expanded to include working with 3D effects such as Digieffects FreeForm plus workflows including Adobe Repouss©, Vanishing Point Exchange, and 3D model import using Adobe Photoshop Extended. The print version is also accompanied by a DVD that contains project files and source materials for all the techniques demonstrated in the book, as well as over 160 pages of bonus chapters on subjects such as expressions, scripting, and effects.

We will be pulling a few “hidden gems” out of each chapter to share on ProVideoCoalition.com roughly every week. These will give you a taste for the multitude of time-saving tips, not-obvious features, little gotchas, and other insider knowledge you will find in CMG.

The content contained in Creating Motion Graphics with After Effects – as well as the CMG Blogs and CMG Keyframes posts on ProVideoCoalition – are copyright Crish Design, except where otherwise attributed.