Ended up using http://videojs.com/ to support all browsers.
But to get the video working in IE9 and Chrome I just added html5 doc type and used mp4:
<!DOCTYPE html>
<html>
<body>
<video src="video.mp4" width="400" height="300" preload controls>
</video>
</body>
</html>