I guess there's no standard HTML5 way to do this, but if you don't mind using a jQuery library, I've been able to achieve a "checkbox group" validation using webshims' "group-required" validation feature:
The docs for group-required say:
If a checkbox has the class 'group-required' at least one of the checkboxes with the same name inside the form/document has to be checked.
And here's an example of how you would use it:
<input name="checkbox-group" type="checkbox" class="group-required" id="checkbox-group-id" />
<input name="checkbox-group" type="checkbox" />
<input name="checkbox-group" type="checkbox" />
<input name="checkbox-group" type="checkbox" />
<input name="checkbox-group" type="checkbox" />
I mostly use webshims to polyfill HTML5 features, but it also has some great optional extensions like this one.
It even allows you to write your own custom validity rules. For example, I needed to create a checkbox group that wasn't based on the input's name, so I wrote my own validity rule for that...