[c#] How I can get web page's content and save it into the string variable

How I can get the content of the web page using ASP.NET? I need to write a program to get the HTML of a webpage and store it into a string variable.

This question is related to c# asp.net screen-scraping

The answer is


I've run into issues with Webclient.Downloadstring before. If you do, you can try this:

WebRequest request = WebRequest.Create("http://www.google.com");
WebResponse response = request.GetResponse();
Stream data = response.GetResponseStream();
string html = String.Empty;
using (StreamReader sr = new StreamReader(data))
{
    html = sr.ReadToEnd();
}

Webclient client = new Webclient();
string content = client.DownloadString(url);

Pass the URL of page who you want to get. You can parse the result using htmlagilitypack.


I recommend not using WebClient.DownloadString. This is because (at least in .NET 3.5) DownloadString is not smart enough to use/remove the BOM, should it be present. This can result in the BOM () incorrectly appearing as part of the string when UTF-8 data is returned (at least without a charset) - ick!

Instead, this slight variation will work correctly with BOMs:

string ReadTextFromUrl(string url) {
    // WebClient is still convenient
    // Assume UTF8, but detect BOM - could also honor response charset I suppose
    using (var client = new WebClient())
    using (var stream = client.OpenRead(url))
    using (var textReader = new StreamReader(stream, Encoding.UTF8, true)) {
        return textReader.ReadToEnd();
    }
}

Examples related to c#

How can I convert this one line of ActionScript to C#? Microsoft Advertising SDK doesn't deliverer ads How to use a global array in C#? How to correctly write async method? C# - insert values from file into two arrays Uploading into folder in FTP? Are these methods thread safe? dotnet ef not found in .NET Core 3 HTTP Error 500.30 - ANCM In-Process Start Failure Best way to "push" into C# array

Examples related to asp.net

RegisterStartupScript from code behind not working when Update Panel is used You must add a reference to assembly 'netstandard, Version=2.0.0.0 No authenticationScheme was specified, and there was no DefaultChallengeScheme found with default authentification and custom authorization How to use log4net in Asp.net core 2.0 Visual Studio 2017 error: Unable to start program, An operation is not legal in the current state How to create roles in ASP.NET Core and assign them to users? How to handle Uncaught (in promise) DOMException: The play() request was interrupted by a call to pause() ASP.NET Core Web API Authentication Could not load file or assembly 'CrystalDecisions.ReportAppServer.CommLayer, Version=13.0.2000.0 WebForms UnobtrusiveValidationMode requires a ScriptResourceMapping for jquery

Examples related to screen-scraping

What's the best way of scraping data from a website? Executing Javascript from Python Can scrapy be used to scrape dynamic content from websites that are using AJAX? How I can get web page's content and save it into the string variable How do I prevent site scraping? Web scraping with Python