The mask editor is a powerful tool to apply masks to video effects. This allows you to create limits, opacities and animations on both layers and video effects/transitions. You can have geometric masks or brush masks.
After you have added a video effect to a part, select it, then select one of the Mask Tools:
Simple mask tool
Line mask tool
Pen mask tool
Brush mask tool
You can also select one of the mask tools after selecting a track, in which case the mask is applied to the entire layer.
The first three tools call one or more geometric masks, while the brush tool creates bitmap brushes. You cannot have both geometric and bitmap brushes at the same time. Geometric masks can transformed and animated through the automation editor, while the bitmap mask allows you to work more precisely by brushing pixels that will or will not mask on a radius and strength parameters.
With the simple mask tool, a mask is automatically created and you can draw it. The top left area of the preview has the "starting" point which you can drag. Then click inside the video preview to create a polygonal mask. This mask will be where the effect will be actually applied.
With the line and pen advanced tools, you have options in the left area (where normally the Bay is displayed):
Add/edit masks. A mask is the time frame of the part that the mask will be applied (by default, it's the entire part, you can resize the bars) and the mask opacity. You can have as many masks as you want. Using multiple masks within different times allows you to create effect animations.
For each mask, add/edit geometries. A geometry is a collection of polygons to apply the mask. If you have more than one geometry, these can be combined to produce the required effect with various ways (unison, inclusion, XOR, exclusion).
For each geometry, add/edit figures. A figure is a polygon (with lines or points, depending on the mask tool selected) which can be filled or hollow and opened or closed. Most times you want it to be filled and closed.
For each figure, you can add items, which are a collection of points you can draw with the mouse. Double clicking on an existing point will remove it.
To animate your mask, you can use either the mask transforms (which transform all the geometries) or per-geometry transforms. The transforms are 2D Effects, (Scale, Skew, Translate, Rotate) and, once you apply a transform to a mask or a geometry, you can animate the parameters through the part with the Automation editor.
The brush tool can help you to create a bitmap opacity brush. Selecting this tool will cancel all geometric masks and present you a drawing interface in which you have drawing tools (rectangle, circle, fill) along with a radius, strength and opacity selector. The more close the opacity is to 1.0 (full), the more the brushed area masks the layer.