When an unauthorized request comes in, the entire request is URL encoded, and added as a query string to the request to the authorization form, so I can see where this may result in a problem given your situation.
According to MSDN, the correct element to modify to reset maxQueryStringLength in web.config is the <httpRuntime>
element inside the <system.web>
element, see httpRuntime Element (ASP.NET Settings Schema). Try modifying that element.