As DotNetMensch said but you DO NOT need to add another site in IIS as this can also cause further problems and make things more complicated because you then have a website within a website so the file paths, masterpage paths and web.config paths may need changing. You just need to edit teh bindings of the existing site and add the new subdomain there.
So:
Add sub-domain to DNS records. My host (RackSpace) uses a web portal to do this so you just log in and go to Network->Domains(DNS)->Actions->Create Zone, and enter your subdomain as mysubdomain.domain.com etc, leave the other settings as default
Go to your domain in IIS, right-click->Edit Bindings->Add, and add your new subdomain leaving everything else the same e.g. mysubdomain.domain.com
You may need to wait 5-10 mins for the DNS records to update but that's all you need.