The only things I can think of that would make a shakey DVD playback on a set-top player is some sort of frame-rate and/or screen dimension conversion, or possibly screen interlace problems.
PAL TV, used in in the UK, is 25 frames per second, and DVDs for PAL use 720x588 pixels. I also think there's a screen interlace. The standard for USA is different, 29.979 (?) frames per second, interlaced and 720x480, and I think the defaults for your software will be to these.
Check the format on your DVD. The set top box(s) may be having to do a lot of conversion and/or not able to properly convert progressive scan (non-interlaced) to interlaced video and this will definitely cause a 'jitter'. Or... you may have produced an interlaced video DVD to USA standards and the fields are reversed going to PAL.
Video is complicated, motion pictures have always been more complicated and making video derived from them is even worse. Frames per second vs pictures per second. Shutter flicker. Then animation woes. Agh!
Fav quote - "Experience is whatcha don't get 'till ya don't need it no more."
System - Athlon 1.4GHz, Win98, Hauppauge PVR250 receiver and compressor.
Software -Magix Movie Edit Pro 10, Nero 6 + NeroVision Express, Moho 4.61, PSP 8.1, Bryce, Quicktime 6.52 pro, Goldwave 5, DVD-Lab.
Cameras - Panasonic GS9, Canon ES8400V, Canon EOS D20 and Canon A70