The information for this is VERY very sparse, so here is a summary of what I have found.

Install libx264

Get the libx264 package from here: https://www.videolan.org/developers/x264.html

Extracts the bz2 file

tar -xjvf /path/to/x264-snapshot-20080805-2245.tar.bz2

And now the useual

cd /path/to/x264-dir
./configure --enable-shared
make
make install
ldconfig

Get a nice error message:

[root@dev01 x264-snapshot-20080805-2245]# ./configure
No suitable assembler found.  Install 'yasm' to get MMX/SSE optimized code.
If you really want to compile without asm, configure with --disable-asm.

You do want MMX/SSE at somepoint for the time being I am disabling this though (lack of time to find a valid RHEL source). so I added the disable asm line.

Install ffmpeg

For this I use subversion.

cd /path/to/where/I/want/sources
svn checkout svn://svn.mplayerhq.hu/ffmpeg/trunk ffmpeg
cd /path/to/where/I/want/sources/ffmpeg
./configure --enable-libx264 --enable-gpl --enable-shared
make
make install

et voila ffmpeg is now installed with libx264 (h264).

error while loading shared libraries: libavdevice.so.52: cannot open shared object file: No such file or directory

To fix this:

vi /etc/ld.so.conf.d/custom-libs.conf

Add the line : /usr/local/lib

And run ldconfig.

UPDATE: I am writing a full set of notes blog entry for installing ffmpeg with codecs on RHEL4