I've found a way to validate a modalpopup without a postback.
In the ModalPopupExtender I set the OnOkScript to a function e.g ValidateBeforePostBack(), then in the function I call Page_ClientValidate for the validation group I want, do a check and if it fails, keep the modalpopup showing. If it passes, I call __doPostBack
.
function ValidateBeforePostBack(){
Page_ClientValidate('MyValidationGroupName');
if (Page_IsValid) { __doPostBack('',''); }
else { $find('mpeBehaviourID').show(); }
}