Let's take an example.
Your application is located in /var/www/myApp on some Linux server
.htaccess : /var/www/myApp/.htaccess
htpasswdApp : /var/www/myApp/htpasswdApp. (You're free to use any name for .htpasswd file)
To use relative path in .htaccess:
AuthType Digest
AuthName myApp
AuthUserFile "htpasswdApp"
Require valid-user
But it will search for file in server_root directory. Not in document_root.
In out case, when application is located at /var/www/myApp :
document_root is /var/www/myApp
server_root is /etc/apache2 //(just in our example, because of we using the linux server)
You can redefine it in your apache configuration file ( /etc/apache2/apache2.conf), but I guess it's a bad idea.
So to use relative file path in your /var/www/myApp/.htaccess you should define the password's file in your server_root.
I prefer to do it by follow command:
sudo ln -s /var/www/myApp/htpasswdApp /etc/apache2/htpasswdApp
You're free to copy my command, use a hard link instead of symbol,or copy a file to your server_root.