You are right. This is a badly documented issue. But you can change the font size parameter (by opposition to font scale) directly after building the plot. Check the following example:
import seaborn as sns
tips = sns.load_dataset("tips")
b = sns.boxplot(x=tips["total_bill"])
b.axes.set_title("Title",fontsize=50)
b.set_xlabel("X Label",fontsize=30)
b.set_ylabel("Y Label",fontsize=20)
b.tick_params(labelsize=5)
sns.plt.show()
, which results in this:
To make it consistent in between plots I think you just need to make sure the DPI is the same. By the way it' also a possibility to customize a bit the rc dictionaries since "font.size" parameter exists but I'm not too sure how to do that.
NOTE: And also I don't really understand why they changed the name of the font size variables for axis labels and ticks. Seems a bit un-intuitive.