It is all but satisfying, isn't it? The easiest way I have found to specify when setting the context, e.g.:
sns.set_context("paper", rc={"font.size":8,"axes.titlesize":8,"axes.labelsize":5})
This should take care of 90% of standard plotting usage. If you want ticklabels smaller than axes labels, set the 'axes.labelsize' to the smaller (ticklabel) value and specify axis labels (or other custom elements) manually, e.g.:
axs.set_ylabel('mylabel',size=6)
you could define it as a function and load it in your scripts so you don't have to remember your standard numbers, or call it every time.
def set_pubfig:
sns.set_context("paper", rc={"font.size":8,"axes.titlesize":8,"axes.labelsize":5})
Of course you can use configuration files, but I guess the whole idea is to have a simple, straightforward method, which is why the above works well.
Note: If you specify these numbers, specifying font_scale in sns.set_context is ignored for all specified font elements, even if you set it.