The situation you describe is pretty fishy. Whenever you close your program's startup form, the entire application should quit automatically, including closing all other open forms. Make sure that you're closing the correct form, and you should not experience any problems.
The other possibility is that you've changed your project (using its Properties page) not to close until all open windows have been closed. In this mode, your application will not exit until the last remaining open form has been closed. If you've chosen this setting, you have to make sure that you call the Close method of all forms that you've shown during the course of application, not just the startup/main form.
The first setting is the default for a reason, and if you've changed it, you probably want to go fix it back.
It is by far the most intuitive model for normal applications, and it prevents exactly the situation you describe. For it to work properly, make sure that you have specified your main form as the "Startup form" (rather than a splash screen or log-in form).
The settings I'm talking about are highlighted here:
But primarily, note that you should never have to call Application.Exit in a properly-designed application. If you find yourself having to do this in order for your program to close completely, then you are doing something wrong. Doing it is not a bad practice in itself, as long as you have a good reason. The other two answers fail to explain that, and thus I feel are incomplete at best.