The contract of the hash code makes clear that:
"If two objects are equal according to the Object method, then calling the hashCode method on each of the two objects must produce the same integer result."
So your assumption:
"To clarify, the equals method is overridden, but it only checks one of the fields, not all. So two Foo objects that are considered equal can actually have different values, that's why I can't just use foo."
is wrong and you are breaking the contract. If we look at the "contains" method of Set interface, we have that:
boolean contains(Object o);
Returns true if this set contains the specified element. More formally, returns true if and only if this set contains an element "e" such that o==null ? e==null : o.equals(e)
To accomplish what you want, you can use a Map where you define the key and store your element with the key that defines how objects are different or equal to each other.