I have downloaded Privoxy few weeks ago and for the fun I was curious to know how a simple version of it can be done.
I understand that I need to configure the browser (client) to send request to the proxy. The proxy send the request to the web (let say it's a http proxy). The proxy will receive the answer... but how can the proxy send back the request to the browser (client)?
I have search on the web for C# and http proxy but haven't found something that let me understand how it works behind the scene correctly. (I believe I do not want a reverse proxy but I am not sure).
Does any of you have some explication or some information that will let me continue this small project?
Update
This is what I understand (see graphic below).
Step 1 I configure the client (browser) for all request to be send to 127.0.0.1 at the port the Proxy listen. This way, request will be not sent to the Internet directly but will be processed by the proxy.
Step2 The proxy see a new connection, read the HTTP header and see the request he must executes. He executes the request.
Step3 The proxy receive an answer from the request. Now he must send the answer from the web to the client but how???
Useful link
Mentalis Proxy : I have found this project that is a proxy (but more that I would like). I might check the source but I really wanted something basic to understand more the concept.
ASP Proxy : I might be able to get some information over here too.
Request reflector : This is a simple example.
Here is a Git Hub Repository with a Simple Http Proxy.
The answer is
Things have become really easy with OWIN and WebAPI. In my search for a C# Proxy server, I also came across this post http://blog.kloud.com.au/2013/11/24/do-it-yourself-web-api-proxy/ . This will be the road I'm taking.
The browser is connected to the proxy so the data that the proxy gets from the web server is just sent via the same connection that the browser initiated to the proxy.
For what it's worth, here is a C# sample async implementation based on HttpListener and HttpClient (I use it to be able to connect Chrome in Android devices to IIS Express, that's the only way I found...).
And If you need HTTPS support, it shouldn't require more code, just certificate configuration: Httplistener with HTTPS support
// define http://localhost:5000 and http://127.0.0.1:5000/ to be proxies for http://localhost:53068
using (var server = new ProxyServer("http://localhost:53068", "http://localhost:5000/", "http://127.0.0.1:5000/"))
{
server.Start();
Console.WriteLine("Press ESC to stop server.");
while (true)
{
var key = Console.ReadKey(true);
if (key.Key == ConsoleKey.Escape)
break;
}
server.Stop();
}
....
public class ProxyServer : IDisposable
{
private readonly HttpListener _listener;
private readonly int _targetPort;
private readonly string _targetHost;
private static readonly HttpClient _client = new HttpClient();
public ProxyServer(string targetUrl, params string[] prefixes)
: this(new Uri(targetUrl), prefixes)
{
}
public ProxyServer(Uri targetUrl, params string[] prefixes)
{
if (targetUrl == null)
throw new ArgumentNullException(nameof(targetUrl));
if (prefixes == null)
throw new ArgumentNullException(nameof(prefixes));
if (prefixes.Length == 0)
throw new ArgumentException(null, nameof(prefixes));
RewriteTargetInText = true;
RewriteHost = true;
RewriteReferer = true;
TargetUrl = targetUrl;
_targetHost = targetUrl.Host;
_targetPort = targetUrl.Port;
Prefixes = prefixes;
_listener = new HttpListener();
foreach (var prefix in prefixes)
{
_listener.Prefixes.Add(prefix);
}
}
public Uri TargetUrl { get; }
public string[] Prefixes { get; }
public bool RewriteTargetInText { get; set; }
public bool RewriteHost { get; set; }
public bool RewriteReferer { get; set; } // this can have performance impact...
public void Start()
{
_listener.Start();
_listener.BeginGetContext(ProcessRequest, null);
}
private async void ProcessRequest(IAsyncResult result)
{
if (!_listener.IsListening)
return;
var ctx = _listener.EndGetContext(result);
_listener.BeginGetContext(ProcessRequest, null);
await ProcessRequest(ctx).ConfigureAwait(false);
}
protected virtual async Task ProcessRequest(HttpListenerContext context)
{
if (context == null)
throw new ArgumentNullException(nameof(context));
var url = TargetUrl.GetComponents(UriComponents.SchemeAndServer, UriFormat.Unescaped);
using (var msg = new HttpRequestMessage(new HttpMethod(context.Request.HttpMethod), url + context.Request.RawUrl))
{
msg.Version = context.Request.ProtocolVersion;
if (context.Request.HasEntityBody)
{
msg.Content = new StreamContent(context.Request.InputStream); // disposed with msg
}
string host = null;
foreach (string headerName in context.Request.Headers)
{
var headerValue = context.Request.Headers[headerName];
if (headerName == "Content-Length" && headerValue == "0") // useless plus don't send if we have no entity body
continue;
bool contentHeader = false;
switch (headerName)
{
// some headers go to content...
case "Allow":
case "Content-Disposition":
case "Content-Encoding":
case "Content-Language":
case "Content-Length":
case "Content-Location":
case "Content-MD5":
case "Content-Range":
case "Content-Type":
case "Expires":
case "Last-Modified":
contentHeader = true;
break;
case "Referer":
if (RewriteReferer && Uri.TryCreate(headerValue, UriKind.Absolute, out var referer)) // if relative, don't handle
{
var builder = new UriBuilder(referer);
builder.Host = TargetUrl.Host;
builder.Port = TargetUrl.Port;
headerValue = builder.ToString();
}
break;
case "Host":
host = headerValue;
if (RewriteHost)
{
headerValue = TargetUrl.Host + ":" + TargetUrl.Port;
}
break;
}
if (contentHeader)
{
msg.Content.Headers.Add(headerName, headerValue);
}
else
{
msg.Headers.Add(headerName, headerValue);
}
}
using (var response = await _client.SendAsync(msg).ConfigureAwait(false))
{
using (var os = context.Response.OutputStream)
{
context.Response.ProtocolVersion = response.Version;
context.Response.StatusCode = (int)response.StatusCode;
context.Response.StatusDescription = response.ReasonPhrase;
foreach (var header in response.Headers)
{
context.Response.Headers.Add(header.Key, string.Join(", ", header.Value));
}
foreach (var header in response.Content.Headers)
{
if (header.Key == "Content-Length") // this will be set automatically at dispose time
continue;
context.Response.Headers.Add(header.Key, string.Join(", ", header.Value));
}
var ct = context.Response.ContentType;
if (RewriteTargetInText && host != null && ct != null &&
(ct.IndexOf("text/html", StringComparison.OrdinalIgnoreCase) >= 0 ||
ct.IndexOf("application/json", StringComparison.OrdinalIgnoreCase) >= 0))
{
using (var ms = new MemoryStream())
{
using (var stream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false))
{
await stream.CopyToAsync(ms).ConfigureAwait(false);
var enc = context.Response.ContentEncoding ?? Encoding.UTF8;
var html = enc.GetString(ms.ToArray());
if (TryReplace(html, "//" + _targetHost + ":" + _targetPort + "/", "//" + host + "/", out var replaced))
{
var bytes = enc.GetBytes(replaced);
using (var ms2 = new MemoryStream(bytes))
{
ms2.Position = 0;
await ms2.CopyToAsync(context.Response.OutputStream).ConfigureAwait(false);
}
}
else
{
ms.Position = 0;
await ms.CopyToAsync(context.Response.OutputStream).ConfigureAwait(false);
}
}
}
}
else
{
using (var stream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false))
{
await stream.CopyToAsync(context.Response.OutputStream).ConfigureAwait(false);
}
}
}
}
}
}
public void Stop() => _listener.Stop();
public override string ToString() => string.Join(", ", Prefixes) + " => " + TargetUrl;
public void Dispose() => ((IDisposable)_listener)?.Dispose();
// out-of-the-box replace doesn't tell if something *was* replaced or not
private static bool TryReplace(string input, string oldValue, string newValue, out string result)
{
if (string.IsNullOrEmpty(input) || string.IsNullOrEmpty(oldValue))
{
result = input;
return false;
}
var oldLen = oldValue.Length;
var sb = new StringBuilder(input.Length);
bool changed = false;
var offset = 0;
for (int i = 0; i < input.Length; i++)
{
var c = input[i];
if (offset > 0)
{
if (c == oldValue[offset])
{
offset++;
if (oldLen == offset)
{
changed = true;
sb.Append(newValue);
offset = 0;
}
continue;
}
for (int j = 0; j < offset; j++)
{
sb.Append(input[i - offset + j]);
}
sb.Append(c);
offset = 0;
}
else
{
if (c == oldValue[0])
{
if (oldLen == 1)
{
changed = true;
sb.Append(newValue);
}
else
{
offset = 1;
}
continue;
}
sb.Append(c);
}
}
if (changed)
{
result = sb.ToString();
return true;
}
result = input;
return false;
}
}
I wouldn't use HttpListener or something like that, in that way you'll come across so many issues.
Most importantly it'll be a huge pain to support:
- Proxy Keep-Alives
- SSL won't work (in a correct way, you'll get popups)
- .NET libraries strictly follows RFCs which causes some requests to fail (even though IE, FF and any other browser in the world will work.)
What you need to do is:
- Listen a TCP port
- Parse the browser request
- Extract Host connect to that host in TCP level
- Forward everything back and forth unless you want to add custom headers etc.
I wrote 2 different HTTP proxies in .NET with different requirements and I can tell you that this is the best way to do it.
Mentalis doing this, but their code is "delegate spaghetti", worse than GoTo :)
Proxy can work in the following way.
Step1, configure client to use proxyHost:proxyPort.
Proxy is a TCP server that is listening on proxyHost:proxyPort. Browser opens connection with Proxy and sends Http request. Proxy parses this request and tries to detect "Host" header. This header will tell Proxy where to open connection.
Step 2: Proxy opens connection to the address specified in the "Host" header. Then it sends HTTP request to that remote server. Reads response.
Step 3: After response is read from remote HTTP server, Proxy sends the response through an earlier opened TCP connection with browser.
Schematically it will look like this:
Browser Proxy HTTP server
Open TCP connection
Send HTTP request ----------->
Read HTTP header
detect Host header
Send request to HTTP ----------->
Server
<-----------
Read response and send
<----------- it back to the browser
Render content
Socks4 is a very simple protocol to implement. You listen for the initial connection, connect to the host/port that was requested by the client, send the success code to the client then forward the outgoing and incoming streams across sockets.
If you go with HTTP you'll have to read and possibly set/remove some HTTP headers so that's a little more work.
If I remember correctly, SSL will work across HTTP and Socks proxies. For a HTTP proxy you implement the CONNECT verb, which works much like the socks4 as described above, then the client opens the SSL connection across the proxied tcp stream.
Proxy can work in the following way.
Step1, configure client to use proxyHost:proxyPort.
Proxy is a TCP server that is listening on proxyHost:proxyPort. Browser opens connection with Proxy and sends Http request. Proxy parses this request and tries to detect "Host" header. This header will tell Proxy where to open connection.
Step 2: Proxy opens connection to the address specified in the "Host" header. Then it sends HTTP request to that remote server. Reads response.
Step 3: After response is read from remote HTTP server, Proxy sends the response through an earlier opened TCP connection with browser.
Schematically it will look like this:
Browser Proxy HTTP server
Open TCP connection
Send HTTP request ----------->
Read HTTP header
detect Host header
Send request to HTTP ----------->
Server
<-----------
Read response and send
<----------- it back to the browser
Render content
I wouldn't use HttpListener or something like that, in that way you'll come across so many issues.
Most importantly it'll be a huge pain to support:
- Proxy Keep-Alives
- SSL won't work (in a correct way, you'll get popups)
- .NET libraries strictly follows RFCs which causes some requests to fail (even though IE, FF and any other browser in the world will work.)
What you need to do is:
- Listen a TCP port
- Parse the browser request
- Extract Host connect to that host in TCP level
- Forward everything back and forth unless you want to add custom headers etc.
I wrote 2 different HTTP proxies in .NET with different requirements and I can tell you that this is the best way to do it.
Mentalis doing this, but their code is "delegate spaghetti", worse than GoTo :)
If you are just looking to intercept the traffic, you could use the fiddler core to create a proxy...
http://fiddler.wikidot.com/fiddlercore
run fiddler first with the UI to see what it does, it is a proxy that allows you to debug the http/https traffic. It is written in c# and has a core which you can build into your own applications.
Keep in mind FiddlerCore is not free for commercial applications.
Agree to dr evil if you use HTTPListener you will have many problems, you have to parse requests and will be engaged to headers and ...
- Use tcp listener to listen to browser requests
- parse only the first line of the request and get the host domain and port to connect
- send the exact raw request to the found host on the first line of browser request
- receive the data from the target site(I have problem in this section)
- send the exact data received from the host to the browser
you see you dont need to even know what is in the browser request and parse it, only get the target site address from the first line first line usually likes this GET http://google.com HTTP1.1 or CONNECT facebook.com:443 (this is for ssl requests)
The browser is connected to the proxy so the data that the proxy gets from the web server is just sent via the same connection that the browser initiated to the proxy.
Socks4 is a very simple protocol to implement. You listen for the initial connection, connect to the host/port that was requested by the client, send the success code to the client then forward the outgoing and incoming streams across sockets.
If you go with HTTP you'll have to read and possibly set/remove some HTTP headers so that's a little more work.
If I remember correctly, SSL will work across HTTP and Socks proxies. For a HTTP proxy you implement the CONNECT verb, which works much like the socks4 as described above, then the client opens the SSL connection across the proxied tcp stream.
Agree to dr evil if you use HTTPListener you will have many problems, you have to parse requests and will be engaged to headers and ...
- Use tcp listener to listen to browser requests
- parse only the first line of the request and get the host domain and port to connect
- send the exact raw request to the found host on the first line of browser request
- receive the data from the target site(I have problem in this section)
- send the exact data received from the host to the browser
you see you dont need to even know what is in the browser request and parse it, only get the target site address from the first line first line usually likes this GET http://google.com HTTP1.1 or CONNECT facebook.com:443 (this is for ssl requests)
I have recently written a light weight proxy in c# .net using TcpListener and TcpClient.
https://github.com/titanium007/Titanium-Web-Proxy
It supports secure HTTP the correct way, client machine needs to trust root certificate used by the proxy. Also supports WebSockets relay. All features of HTTP 1.1 are supported except pipelining. Pipelining is not used by most modern browsers anyway. Also supports windows authentication (plain, digest).
You can hook up your application by referencing the project and then see and modify all traffic. (Request and response).
As far as performance, I have tested it on my machine and works without any noticeable delay.
If you are just looking to intercept the traffic, you could use the fiddler core to create a proxy...
http://fiddler.wikidot.com/fiddlercore
run fiddler first with the UI to see what it does, it is a proxy that allows you to debug the http/https traffic. It is written in c# and has a core which you can build into your own applications.
Keep in mind FiddlerCore is not free for commercial applications.
Things have become really easy with OWIN and WebAPI. In my search for a C# Proxy server, I also came across this post http://blog.kloud.com.au/2013/11/24/do-it-yourself-web-api-proxy/ . This will be the road I'm taking.
I have recently written a light weight proxy in c# .net using TcpListener and TcpClient.
https://github.com/titanium007/Titanium-Web-Proxy
It supports secure HTTP the correct way, client machine needs to trust root certificate used by the proxy. Also supports WebSockets relay. All features of HTTP 1.1 are supported except pipelining. Pipelining is not used by most modern browsers anyway. Also supports windows authentication (plain, digest).
You can hook up your application by referencing the project and then see and modify all traffic. (Request and response).
As far as performance, I have tested it on my machine and works without any noticeable delay.
Source: Stackoverflow.com