Default Mplayer installation autodetects Xvid, H264 and other codecs. So if you have Xvid and H264 codecs installed your mencoder and mplayer should support it, unless you disable them during compilation. In order to have your Mencoder and Mplayer support Xvid and H264 you need to install them first and then compile/re-compile Mencoder/Mplayer.
H264/x264 is a library for creating H.264 video. MPlayer sources are updated whenever an x264 API change occurs, so it is always suggested to use MPlayer from Subversion. At this time of post x264 in on development release only.