I can confirm that mp4 just will not work in the video tag. No matter how much you try to mess with the type tag and the codec and the mime types from the server.
Crazy, because for the same exact video, on the same test page, the old embed tag for an mp4 works just fine in firefox. I spent all yesterday messing with this. Firefox is like IE all of a sudden, hours and hours of time, not billable. Yay.
Speaking of IE, it fails FAR MORE gracefully on this. When it can't match up the format it falls to the content between the tags, so it is possible to just put video around object around embed and everything works great. Firefox, nope, despite failing, it puts up the poster image (greyed out so that isn't even useful as a fallback) with an error message smack in the middle. So now the options are put in browser recognition code (meaning we've gained nothing on embedding videos in the last ten years) or ditch html5.