I just ran into this the other day.
What I did, which seems easier than the above, is to set the pattern on a variable on the scope and refer to it in ng-pattern in the view.
When "the checkbox is unchecked" I simply set the regex value to /.*/ on the onChanged callback (if going to unchecked). ng-pattern picks that change up and says "OK, your value is fine". Form is now valid. I would also remove the bad data from the field so you don't have an apparent bad phone # sitting there.
I had additional issues around ng-required, and did the same thing. Worked like a charm.