Mmm first your id attributes must be unique, your code is likely to be
<form>
<input class='roles' name='roles' type='checkbox' value='1' />
<input class='roles' name='roles' type='checkbox' value='2' />
<input class='roles' name='roles' type='checkbox' value='3' />
<input class='roles' name='roles' type='checkbox' value='4' />
<input class='roles' name='roles' type='checkbox' value='5' />
<input type='submit' value='submit' />
</form>
For your problem :
if($('.roles:checkbox:checked').length == 0)
// no checkbox checked, do something...
else
// at least one checkbox checked...
BUT, remember that a JavaScript form validation is only indicative, all validations MUST be done server-side.