Had similar problems recently. Would suggest you carefully check if the user you're connecting with has proper authorizations on the remote machine.
You can review permissions using the following command.
Set-PSSessionConfiguration -ShowSecurityDescriptorUI -Name Microsoft.PowerShell
Found this tip here (updated link, thanks "unbob"):
https://devblogs.microsoft.com/scripting/configure-remote-security-settings-for-windows-powershell/
It fixed it for me.