You should always include the table name in the SQL query when dealing with associations.
Indeed if another table has the user_id column and you join both tables, you will have an ambiguous column name in the SQL query (i.e. troubles).
So, in your example:
GroupUser.where("groups_users.user_id != ?", me)
Or a bit more verbose:
GroupUser.where("#{table_name}.user_id IS NOT ?", me)
Note that if you are using a hash, you don't need to worry about that because Rails takes care of it for you:
GroupUser.where(user: me)
In Rails 4, as said by @dr4k3, the query method not has been added:
GroupUser.where.not(user: me)