



In After Effects, each mask vertex (the yellow squares) interpolates in a straight line from old to new positions. The red lines illustrate some of these paths. The First Vertex is the largest box – in this case, the one in the upper left corner of the M and I.
Masking is one of the core features of After Effects. Most know how to create and edit Mask Shapes; fewer how to control the way these shapes animate – which is important, especially with the popularity of creating cel or Flash type animations these days.
After Effects can seem to have a twisted mind of its own when interpolating between two different Mask Shapes. In reality, it has a very narrow, simple mind. However, there are some tricks you can employ to coax it down a path closer to the one you want. When you need even more precise control, you can employ the Smart Mask Interpolation keyframe assistant, included in the Professional edition of After Effects (including CS3 Professional).
Vertices Unmasked
When you create a Mask Shape in After Effects, the points that defines the mask’s outline are known as the its vertices. When a Mask Shape is keyframed, After Effects attempts to interpolate between shapes by making each vertex travel in a straight line from its old to its new position. Refer to the figures at the top of this page to see this in action - the red lines have been added to better illustrate the movement of a handful of the vertices.
Those familiar with morphing software can think of each vertex as a “correspondence point” which would normally be set up by the user to ensure the outline kept its desired shape during interpolation. However, After Effects sets up these correspondences for you automatically. If you aren’t aware of these correspondences, complex mask shapes can seem to have a mind of their own as they interpolate, rotating and inverting in unexpected ways.
The most important mask point is the First Vertex. This is where you started the mask shape when you created it. After Effects always interpolates from the First Vertex of the first shape to the First Vertex of the second shape, regardless of how many other vertices exist or what order they were drawn in. If the correspondence between these vertices doesn’t make any sense - for example, at the top of one shape and at the bottom of the other - your mask shape is going to twist and turn in usually undesirable ways between keyframes.
The First Vertex can be identified by selecting the mask, and looking for the vertex box that is ever so slightly larger than all of the other ones. To change it, select any other vertex (with no other vertices selected), and use the menu command Layer > Mask > Set First Vertex. You can also context-click on it and select Set First Vertex from the menu that appears.
Setting up good First Vertex points is the single most important thing you can do to ensure masks interpolate more as you expect. To get smoother interpolation, study your mask shapes and look for a good “anchor” the shapes have in common, such as the center point along their top or bottom. Make these points your First Vertex. If you need to add mask vertices to your shapes to create these more suitable anchors, go ahead!
After looking at the First Vertex, After Effects will then attempt to line up additional vertex points, working clockwise around the mask shape. If the mask shape was imported from elsewhere - say, from an Adobe Illustrator outline - it is possible that the vertex points are numbered in a counterclockwise direction, which will make shapes appear to invert (turn inside-out) as they animate.
After Effects does not allow you to directly change a mask path’s direction, but there are a couple of workarounds. The time-honored one is to use Illustrator to change it. In After Effects, you will need to select just the keyframe for the mask shape (not the shape itself, nor the entire layer). Copy, and then paste this shape into an Illustrator document. Then press Command+8 on Mac (Control+8 on Windows) in Illustrator to convert it into a Compound Shape. If needed, use the Reverse Path Direction buttons in Illustrator’s Attributes palette (although the simple act of enabling Compound Paths in Illustrator 10 will give you the reversed result). Then copy the shape in Illustrator, return to After Effects, make sure the Mask Shape keyframe is still selected, and paste.
If your First Vertex or Path Direction are set up wrong, nothing can help you - not even the Professional version’s Smart Mask Interpolation (SMI) keyframe assistant. Fixing the vertices and path direction are often enough to get mask shapes to behave. But if you need more control, then Smart Mask Interpolation is the tool you need.
(Page 1 of 3 pages for this article 1 2 3 >)
Name:
Email:
Location:
URL:
Remember my personal information
Notify me of follow-up comments?
Submit the word you see below:

