I haven't had to re-size a menu before, but I have used a free program called PGCedit to mess around with a finished DVD's structure (I assume this will run on a Mac aswell), the site for it is here:
You'll need to copy your DVD over to the Hard drive so you have the VIDEO_TS and AUDIO_TS folders available for PGCedit to access.
and here's a guide that describes how to edit a menu's buttons, which could probably be applied to editing the whole menu:
If that doesn't help, you could try searching Doom9.org for a solution.
Panasonic SD9, Panasonic NVGS-75, Canon MV600i