If you are using Debian style virtual host configuration (sites-available/sites-enabled), one way to set a Default VirtualHost is to include the specific configuration file first in httpd.conf or apache.conf (or what ever is your main configuration file).
# To set default VirtualHost, include it before anything else.
IncludeOptional sites-enabled/my.site.com.conf
# Load config files in the "/etc/httpd/conf.d" directory, if any.
IncludeOptional conf.d/*.conf
# Load virtual host config files from "/etc/httpd/sites-enabled/".
IncludeOptional sites-enabled/*.conf