I had been running into this issue in Vegas 11, so I tried Vegas 12 and am running into the same issue. When I render a project that has an mp4 file included in it, Vegas gradually eats up memory until there is none free and the program crashes. I am rendering to WMV, 720. It has to be mp4 related, as I do not have this issue when any other file types are included in my project.
Here are my system specs:
Win7 Ultimate 64
Core i7 3.4
I'm sure there is more information that you would need to help determine a solution - please feel free to ask. This is slowly driving me mad. I've messed around with the dynamic RAM and rendering threads (as many other sites claim this will solve the problem), but most places attribute this issue to 32-bit limitations, and that's obviously not the case here.
I've also tried converting the mp4 into something else first, but that doesn't always work well and, to be honest, it doesn't really solve the problem. Also, just to put this out there - yes, I have to render into WMV.