After five years, I can see this is still a very relevant subject. Somewhat mirroring leucos's answer which I find the best in my case, using ansible tools only (without any centralised authentication, tokens or whatever). This assumes you have the same username and the same public key on all servers. If you don't, of course you'd need to be more specific and add the corresponding variables next to the hosts:
[all:vars]
ansible_ssh_user=ansible
ansible_ssh_private_key_file=home/user/.ssh/mykey
[group]
192.168.0.50 ansible_sudo_pass='{{ myserver_sudo }}'
ansible-vault create mypasswd.yml
ansible-vault edit mypasswd.yml
Add:
myserver_sudo: mysecretpassword
Then:
ansible-playbook -i inv.ini my_role.yml --ask-vault --extra-vars '@passwd.yml'
At least this way you don't have to write more the variables which point to the passwords.