The exception that we ran into was not on the local but on the remote server, the Azure CI was reading it from the packages folder but the compiler versions mentioned above were not found.
To fix this we modified the project file to make it something like
It is not referencing any of the packages here directly referencing the environment variables.
This fixed the issue, however in our cases we don't use packages directly from "package.config" instead we have a separate folder to maintain version integrity across teams.