[.net] .Net HttpWebRequest.GetResponse() raises exception when http status code 400 (bad request) is returned

I am in a situation where when I get an HTTP 400 code from the server, it is a completely legal way of the server telling me what was wrong with my request (using a message in the HTTP response content)

However, the .NET HttpWebRequest raises an exception when the status code is 400.

How do I handle this? For me a 400 is completely legal, and rather helpful. The HTTP content has some important information but the exception throws me off my path.

I know this has already been answered a long time ago, but I made an extension method to hopefully help other people that come to this question.

Code:

public static class WebRequestExtensions
{
    public static WebResponse GetResponseWithoutException(this WebRequest request)
    {
        if (request == null)
        {
            throw new ArgumentNullException("request");
        }

        try
        {
            return request.GetResponse();
        }
        catch (WebException e)
        {
            if (e.Response == null)
            {
                throw;
            }

            return e.Response;
        }
    }
}

Usage:

var request = (HttpWebRequest)WebRequest.CreateHttp("http://invalidurl.com");

//... (initialize more fields)

using (var response = (HttpWebResponse)request.GetResponseWithoutException())
{
    Console.WriteLine("I got Http Status Code: {0}", response.StatusCode);
}

Try this (it's VB-Code :-):

Try

Catch exp As WebException
  Dim sResponse As String = New StreamReader(exp.Response.GetResponseStream()).ReadToEnd
End Try

I had similar issues when trying to connect to Google's OAuth2 service.

I ended up writing the POST manually, not using WebRequest, like this:

TcpClient client = new TcpClient("accounts.google.com", 443);
Stream netStream = client.GetStream();
SslStream sslStream = new SslStream(netStream);
sslStream.AuthenticateAsClient("accounts.google.com");

{
    byte[] contentAsBytes = Encoding.ASCII.GetBytes(content.ToString());

    StringBuilder msg = new StringBuilder();
    msg.AppendLine("POST /o/oauth2/token HTTP/1.1");
    msg.AppendLine("Host: accounts.google.com");
    msg.AppendLine("Content-Type: application/x-www-form-urlencoded");
    msg.AppendLine("Content-Length: " + contentAsBytes.Length.ToString());
    msg.AppendLine("");
    Debug.WriteLine("Request");
    Debug.WriteLine(msg.ToString());
    Debug.WriteLine(content.ToString());

    byte[] headerAsBytes = Encoding.ASCII.GetBytes(msg.ToString());
    sslStream.Write(headerAsBytes);
    sslStream.Write(contentAsBytes);
}

Debug.WriteLine("Response");

StreamReader reader = new StreamReader(sslStream);
while (true)
{  // Print the response line by line to the debug stream for inspection.
    string line = reader.ReadLine();
    if (line == null) break;
    Debug.WriteLine(line);
}

The response that gets written to the response stream contains the specific error text that you're after.

In particular, my problem was that I was putting endlines between url-encoded data pieces. When I took them out, everything worked. You might be able to use a similar technique to connect to your service and read the actual response error text.

This solved it for me:
https://gist.github.com/beccasaurus/929007/a8f820b153a1cfdee3d06a9c0a1d7ebfced8bb77

TL;DR:
Problem:
localhost returns expected content, remote IP alters 400 content to "Bad Request"
Solution:
Adding <httpErrors existingResponse="PassThrough"></httpErrors> to web.config/configuration/system.webServer solved this for me; now all servers (local & remote) return the exact same content (generated by me) regardless of the IP address and/or HTTP code I return.

Interestingly, the HttpWebResponse.GetResponseStream() that you get from the WebException.Response is not the same as the response stream that you would have received from server. In our environment, we're losing actual server responses when a 400 HTTP status code is returned back to the client using the HttpWebRequest/HttpWebResponse objects. From what we've seen, the response stream associated with the WebException's HttpWebResponse is generated at the client and does not include any of the response body from the server. Very frustrating, as we want to message back to the client the reason for the bad request.

An asynchronous version of extension function:

    public static async Task<WebResponse> GetResponseAsyncNoEx(this WebRequest request)
    {
        try
        {
            return await request.GetResponseAsync();
        }
        catch(WebException ex)
        {
            return ex.Response;
        }
    }