How to save webpage to string with cookies support (httpWebRequest)

Posted by Maciej on Stack Overflow See other posts from Stack Overflow or by Maciej
Published on 2010-04-06T13:04:08Z Indexed on 2010/04/06 13:13 UTC
Read the original article Hit count: 592

Filed under:
|
|
|

I need to read webpage and store its content in string for further processing.
Sounds simply but I have problem with cookies support.

Opened page says I need browser supporting cookies (or turned on).
I've made method trying do that via httpWebRequest - which normally works to me but I've come to a standstill with those unfortunate cookies...

Any idea how to make it working?

Here is my method:

string ReadHtml (string address, string encoding) {
   Uri url = new Uri(address);

   CookieContainer cookieContainer  = new CookieContainer();
            HttpWebRequest httpWebRequest  =     (HttpWebRequest)HttpWebRequest.Create(url);
   httpWebRequest.AllowAutoRedirect = true;
            httpWebRequest.KeepAlive   = true;
            httpWebRequest.CookieContainer  = cookieContainer;
            httpWebRequest.UserAgent   = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)";
            httpWebRequest.Method    = "GET";
            HttpWebResponse webResponse = (HttpWebResponse)httpWebRequest.GetResponse();

   // Code Page
   Encoding enc = Encoding.GetEncoding(encoding);  

   // Read content
   StreamReader loResponseStream = new     StreamReader(webResponse.GetResponseStream(),enc);
   string   lcHtml    = loResponseStream.ReadToEnd();

   webResponse.Close();
   loResponseStream.Close();

   return lcHtml;

  }

© Stack Overflow or respective owner

Related posts about c#

Related posts about .NET