video/mp4should be used when you have video content in your file. If there is none, but there is audio, you should use audio/mp4. If no audio and no video is used, for instance if the file contains only a subtitle track or a metadata track, the MIME should be application/mp4.
Also, as a server, you should try to include the codecs or profiles parameters as defined in RFC6381, as this will help clients determine if they can play the file, prior to downloading it.