I have come to this page multiple times to get the answer, and while all the existing answers were super helpful, they did not solve my problem exactly.
google.maps.event.addListenerOnce(googleMap, 'zoom_changed', function() {
var oldZoom = googleMap.getZoom();
googleMap.setZoom(oldZoom - 1); //Or whatever
});
Basically I found that the 'zoom_changed' event prevented the UI of the map from "skipping" which happened when i waited for the 'idle' event.
Hope this helps somebody!