I'd like to clarify one small gotcha here. You must use CustomVideoMode1 before CustomVideoMode2, etc. VirtualBox recognizes these modes in order starting from 1 and if you skip a number, it will not recognize anything at or beyond the number you skipped. This caught me by surprise.