While refresh token is retained by the Authorization server. Access token are self-contained so resource server can verify it without storing it which saves the effort of retrieval in case of validation. Another point missing in discussion is from rfc6749#page-55
"For example, the authorization server could employ refresh token rotation in which a new refresh token is issued with every access token refresh response.The previous refresh token is invalidated but retained by the authorization server. If a refresh token is compromised and subsequently used by both the attacker and the legitimate client, one of them will present an invalidated refresh token, which will inform the authorization server of the breach."
I think the whole point of using refresh token is that even if attacker somehow manages to get refresh token, client ID and secret combination. With subsequent calls to get new access token from attacker can be tracked in case if every request for refresh result in new access token and refresh token.