The invalid hostname indicates that the actual site you configured in the IIS Express configuration file is (most likely) not running. IIS Express doesn't have a process model like IIS does.
For your site to run it would need to be started explicitly (either by opening and accessing from webmatrix, or from command line calling iisexpress.exe (from it's installation directory) with the /site parameter.
In general, the steps to allow fully qualified DNS names to be used for local access are Let's use your example of the DNS name dev.example.com
Configure IIS Express binding for your site (eg:Site1) to include dev.example.com. Administrative privilege will be needed to use the binding. Alternatively, a one-time URL reservation can be made with http.sys using
netsh http add urlacl url=http://dev.example.com:<port>/ user=<user_name>
start iisexpress /site:Site1
or open Site1 in WebMatrix