CMG Hidden Gems: Chapter 10 – All About Masking

As you are probably aware by now, we’re going through our book Creating Motion Graphics with After Effects 5th Edition (“CMG5” for short) and pulling out a few “hidden gems” from each chapter. These will include both essential tips for new users, as well as timesaving gems that experienced users may not be aware existed.

Chapter 10 covers creating and editing simple geometric mask shapes, Bezier masks, and RotoBezier masks. We build on your expertise until you’re comfortable managing multiple masks with Mask Modes. We also discuss using effects with masks, rotoscoping techniques, importing paths from Photoshop and Illustrator, as well as the Mask Interpolation keyframe assistant.

Viewing Outside the Mask

Footage courtesy Artbeats/Lifestyles Mixed Cuts 1.

While you can draw masks directly in the Comp panel, you can’t see the area of the image that you’ve masked out. But you can in the Layer panel – if you set up the Layer panel options correctly.

In the figure above, note the View menu and the Render checkbox in the Layer panel (left viewer). These decide what part of the rendering chain is displayed. When the Render switch is off, you can see the entire Source in the Layer panel and the results of your masking in the Comp panel (right viewer).

Remember that to quickly switch to this split view, select the Comp panel and press Command+Option+Shift+N on Mac (Control+Alt+Shift+N on Windows) to get two viewers side by side. Now you can use the left viewer for the Layer panel.

Shape Shortcuts

There are a number of ways you can manipulate the basic shapes created with the Shape tools while you’re initially drawing a mask. Below are just a few:

• Double-click the Shape tools icon to create a mask that uses the currently active shape (Rectangle, Ellipse, Star, and so forth) and that has the same outer dimensions as the layer. (If another mask is selected when you double-click the Shape tool, that mask will be replaced.)
• The Polygon and Star tools always draw from the center out; Rectangle, Rounded Rectangle, and Ellipse draw from a corner out. To draw rectangles or ellipses from the center out, press Command (Control) after you start dragging.
• To interactively change the mask you are drawing with the Shape tool, there are a set of modifier keys you can press after you start drawing the shape but before you release the mouse. For instance, if you draw a Rounded Rectangle mask shape (see figure below), before you release the mouse, press the up/down cursor keys to increase/decrease the corner roundness. This same shortcut also edits the number of side on a Polygon shape or number of points on a Star shape. [Note that the same shortcuts will also work when creating Shape Layers (Chapter 32), but with shape layer you can always edit the shapes numerically in the timeline after the fact – a luxury you don’t get when creating masks shapes!]

Rotobezier Masks

An alternative to working with Bezier handles is to create RotoBezier masks. This type of mask – also created using the Pen tool – automatically calculates the curves leading into and out of mask points based on the positions of adjacent points. RotoBezier shapes can be easier to use for flowing, organic objects that feature curves rather than hard corners.

To create a RotoBezier mask, select a layer, then select the Pen tool. Along the Tools panel to the right the word RotoBezier will appear; enable it by checking its box (see figure).

As you click, you’ll initially see a straight line segment between your two most recent points; as you continue adding points, After Effects calculates the curves between the earlier points. Don’t sweat it if you’re unhappy with the immediate results; you can edit your points later.

After you have finished drawing a mask, return to the Selection tool. Click and drag any point around, and watch as the curves before and after it automatically bend to follow.

You can adjust the otherwise-automatic tension of RotoBezier masks either after they’re created or while you are drawing. Hold down Command+Option (Control+Alt) and click on a mask point to change it to a hard corner; doing so again returns its tension to 25%. Clicking and dragging with these modifier keys held allows you to adjust the tension of the imaginary Bezier handles running through the mask point; the Info panel updates the value between 0% and 100% as you drag. Note that these commands also work on all selected points, so be aware of what you have selected!

Holding Command+Option (Control+Alt) allows you to adjust the tension (roundness) of a RotoBezier point.
Footage courtesy Artbeats/Incarcerated.

First Vertex Point

When you select all points on a mask, have you noticed that one point is a bit larger than all the others? This is the First Vertex Point, and it controls how mask points interpolate. The First Vertex Point at one keyframe always interpolates to the First Vertex Point on the next keyframe – all the other points around the path follow their lead. When you’re interpolating from very different shapes, you may need to manually change this “leader” point to create the best match from shape to shape. To do so, select the mask point you want to use as the First Vertex, right-click on it, and select Mask > Set First Vertex.

Note: The First Vertex Point is also the point used for the start/end of a closed mask shape when it’s used by an effect (such as Stroke) and when creating text on a circle (or any closed shape).

Locking & Hiding Masks

When you’re working with multiple masks, you can lock a mask to protect it from accidentally being edited. Just click the Lock box associated with each mask group in the A/V Features column. Masks will still appear in the Comp and Layer panels, but you won’t be able to edit them or change any keyframes.

To hide locked masks from displaying, select Mask > Hide Locked Masks.

Keeping masks in sync with the source

A mask is drawn to fit a movie running at 24 fps (upper left). When nested into a comp running at 29.97 fps, the mask will look correct only when it lines up with a keyframe (upper right). Interpolated frames will appear to slip (left).
Pendulum courtesy Artbeats/Time & Money.

If a keyframed mask shape appears to be “slipping” position in the rendered movie, even though it looks fine in the composition, there’s a likely culprit: A mask is interpolating out of sync with the source frames. This is caused by a mismatch between the mask’s keyframes and the frame (or field) rate of your footage. For example, if your footage has 24 progressive frames (not interlaced frames), you would mask this layer on whole frames in a 24 fps composition. However, if you then nest this precomp in the final comp and render at 29.97 fps, the mask will now interpolate at 29.97 fps (or at 59.94 fps, of you field render!) and will appear to slip in between frames of the original source.

To solve this problem, you can enable the Preserve Frame Rate option in the precomps Composition Settings > Advanced tab. Now the nested comp’s frame rate will be honored no matter what frame rate the final comp is rendered at. (Nesting compositions and the Preserve Frame Rate option are covered in Chapter 18.)

Trish and Chris Meyer share seventeen-plus years of real-world film and video production experience inside their now-classic book Creating Motion Graphics with After Effects (CMG).

The 5th edition has been thoroughly revised to reflect the new features introduced in both After Effects CS4 and CS5. New chapters cover the new Roto Brush feature, as well as mocha and mocha shape. The 3D section has been expanded to include working with 3D effects such as Digieffects FreeForm plus workflows including Adobe Repouss©, Vanishing Point Exchange, and 3D model import using Adobe Photoshop Extended. The print version is also accompanied by a DVD that contains project files and source materials for all the techniques demonstrated in the book, as well as over 160 pages of bonus chapters on subjects such as expressions, scripting, and effects.

We will be pulling a few “hidden gems” out of each chapter to share on ProVideoCoalition.com roughly every week. These will give you a taste for the multitude of time-saving tips, not-obvious features, little gotchas, and other insider knowledge you will find in CMG.

The content contained in Creating Motion Graphics with After Effects – as well as the CMG Blogs and CMG Keyframes posts on ProVideoCoalition – are copyright Crish Design, except where otherwise attributed.