Maybe I am off the mark here and not understanding the OP but why are you joining tables?
If you have a table with members and this table has a column named "group_id", you can just run a query on the members table to get a count of the members grouped by the group_id.
SELECT group_id, COUNT(*) as membercount
FROM members
GROUP BY group_id
HAVING membercount > 4
This should have the least overhead simply because you are avoiding a join but should still give you what you wanted.
If you want the group details and description etc, then add a join from the members table back to the groups table to retrieve the name would give you the quickest result.