How to save webpage to string with cookies support (httpWebRequest)
- by Maciej
I need to read webpage and store its content in string for further processing.
Sounds simply but I have problem with cookies support.
Opened page says I need browser supporting cookies (or turned on).
I've made method trying do that via httpWebRequest - which normally works to me but I've come to a standstill with those unfortunate cookies...
Any idea how to make it working?
Here is my method:
string ReadHtml (string address, string encoding) {
Uri url = new Uri(address);
CookieContainer cookieContainer = new CookieContainer();
HttpWebRequest httpWebRequest = (HttpWebRequest)HttpWebRequest.Create(url);
httpWebRequest.AllowAutoRedirect = true;
httpWebRequest.KeepAlive = true;
httpWebRequest.CookieContainer = cookieContainer;
httpWebRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)";
httpWebRequest.Method = "GET";
HttpWebResponse webResponse = (HttpWebResponse)httpWebRequest.GetResponse();
// Code Page
Encoding enc = Encoding.GetEncoding(encoding);
// Read content
StreamReader loResponseStream = new StreamReader(webResponse.GetResponseStream(),enc);
string lcHtml = loResponseStream.ReadToEnd();
webResponse.Close();
loResponseStream.Close();
return lcHtml;
}
