dude you need ffmpegx for osx. Its a biit odd to install as you need to download its libraries and encoders from different websites it links you to but its worth taking the time to do. I converted an unencrypted movie that was a few vob's to a atsc mpeg2 and enjoyed it being served from my powerbook just last night. That was with quicktime pro doing the encoding. I am going to try sorensons squeeze and cleaner for mac this evening as they are known for cleaner video encoding.
hope it helps
self proclaimed enthusiast of bright shiney objects