You most certainly can (at least in VMS6)
Video on lower track.
Mask on track above (white is the are you want to show through, black is the area you don't want to show, greys are in between)
Hover over the green icon at the right hand side of the track header of the "mask" track - it'll say "Compositing mode" - click and select "Mask"
Enough info for you to be getting on with?