Go on using tabTextAppearance as you did but
1) to fix the capital letter side effect add textAllCap in your style :
<style name="MyTabLayoutTextAppearance" parent="TextAppearance.AppCompat.Widget.ActionBar.Title.Inverse">
<item name="android:textSize">14sp</item>
<item name="android:textAllCaps">true</item>
</style>
2) to fix the selected tab color side effect add in TabLayout xml the following library attributes :
app:tabSelectedTextColor="@color/color1"
app:tabTextColor="@color/color2"
Hope this helps.