Turns out this was a problem, because the ASP.NET MVC 4 project was referencing a specific version of the Microsoft.Net.Compilers package. Visual Studio was using the compiler from this specific package, and not the compiler that was installed otherwise on the computer.
A warning or something would have been nice from VS2019 :-)
The solution then is to update the Microsoft.Net.Compilers package to a newer version.
Version 1.x is for C# 6 Version 2.x is for C# 7 Version 3.x is for C# 8 How I got to solve this was not immediately obvious. Visual Studio could have suggested or hinted that by me selecting a new version in the project settings that setting now conflicted with the package installed into the project.
(I ended up turning on Diagnostics level MSBuild logging to find out which CSC.EXE the IDE is really trying to use)