I actually looked at this a little in the disassembler, since source isn't available.
/usr/bin/java and /usr/libexec/java_home both make use of JavaLaunching.framework. The JAVA_HOME environment variable is indeed checked first by /usr/bin/java and friends (but not /usr/libexec/java_home.) The framework uses the JAVA_VERSION and JAVA_ARCH envirionment variables to filter the available JVMs. So, by default:
$ /usr/libexec/java_home -V
Matching Java Virtual Machines (2):
11.0.5, x86_64: "Amazon Corretto 11" /Library/Java/JavaVirtualMachines/amazon-corretto-11.jdk/Contents/Home
1.8.0_232, x86_64: "Amazon Corretto 8" /Library/Java/JavaVirtualMachines/amazon-corretto-8.jdk/Contents/Home
/Library/Java/JavaVirtualMachines/amazon-corretto-11.jdk/Contents/Home
But setting, say, JAVA_VERSION can override the default:
$ JAVA_VERSION=1.8 /usr/libexec/java_home
/Library/Java/JavaVirtualMachines/amazon-corretto-8.jdk/Contents/Home
You can also set JAVA_LAUNCHER_VERBOSE=1 to see some additional debug logging as far as search paths, found JVMs, etc., with both /usr/bin/java and /usr/libexec/java_home.
In the past, JavaLaunching.framework actually used the preferences system (under the com.apple.java.JavaPreferences domain) to set the preferred JVM order, allowing the default JVM to be set with PlistBuddy - but as best as I can tell, that code has been removed in recent versions of macOS. Environment variables seem to be the only way (aside from editing the Info.plist in the JDK bundles themselves.)
Setting default environment variables can of course be done through your .profile or via launchd, if you need them be set at a session level.