Any user with a valid shell in /etc/passwd can potentially login. If you want to improve security, set up SSH with public-key authentication (there is lots of info on the web on doing this), install a public key in one user's ~/.ssh/authorized_keys file, and disable password-based authentication. This will prevent anybody except that one user from logging in, and will require that the user have in their possession the matching private key. Make sure the private key has a decent passphrase.
To prevent bots from trying to get in, run SSH on a port other than 22 (i.e. 3456). This doesn't improve security but prevents script-kiddies and bots from cluttering up your logs with failed attempts.