If the variable ax.xaxis._autolabelpos = True, matplotlib sets the label position in function _update_label_position in axis.py according to (some excerpts):
bboxes, bboxes2 = self._get_tick_bboxes(ticks_to_draw, renderer)
bbox = mtransforms.Bbox.union(bboxes)
bottom = bbox.y0
x, y = self.label.get_position()
self.label.set_position((x, bottom - self.labelpad * self.figure.dpi / 72.0))
You can set the label position independently of the ticks by using:
ax.xaxis.set_label_coords(x0, y0)
that sets _autolabelpos to False or as mentioned above by changing the labelpad parameter.