I came across this when I wanted to make the axis labels smaller, but leave everything else the same size. The command that worked for me, was to put:
par(cex.axis=0.5)
Before the plot command. Just remember to put:
par(cex.axis=1.0)
After the plot to make sure that the fonts go back to the default size.