Ya idk. I was tring to figure it out on my mac, but i dont know what to convert it to.
For the Mac I'd use a program called FFmpegx. It puts a nice front end on FFmpeg and can encode just about any format out there.
http://www.ffmpegx.com/I run Linux and use FFmpeg from the cli and find it very thorough with a learning curve for the advanced features.
As for what to convert it to, I use Xvid for video and AAC for audio then stuff it into an mp4 container. I've achieved nice results with 2 pass encoding for video with a bitrate between 800-1000kb and audio at 160kb stereo.
A couple of things I've learned, the screen resolution is 432x240, the Instinct uses a 18bpp colour space and has to dither to simulate 24bpp colour, encoding with B frames will make the phone puke, Qsel and GMC are processor intensive and probably will make the phone puke. The Iphone has a AAC encoding limit of 128kb