$setValidity needs to be called on the ngModelController. Inside the controller, I think that means $scope.myForm.file.$setValidity()
.
See also section "Custom Validation" on the Forms page, if you haven't already.
Also, for the first argument to $setValidity, use just 'filetype' and 'size'.