C# WebClient only downloads partial html

Posted by H4mm3rHead on Stack Overflow See other posts from Stack Overflow or by H4mm3rHead
Published on 2010-04-13T09:35:57Z Indexed on 2010/04/13 9:42 UTC
Read the original article Hit count: 463

Filed under:
|

Hi, I am working on some scraping app, i wanted to try to get it to work but ran into a problem. I have replaced the original scraping destination in the below code with googles webpage, just for testing. It seems that my download doesnt get everything, i note that the body and the html tags are missing their close tags. How do i get it to download everything? Whats wrong with my sample code:

string filename = "test.html";

WebClient client = new WebClient();            
string searchTerm = HttpUtility.UrlEncode(textBox2.Text);            
client.QueryString.Add("q", searchTerm);
client.QueryString.Add("hl", "en");
string data = client.DownloadString("http://www.google.com/search");

StreamWriter writer = new StreamWriter(filename, false, Encoding.Unicode);
writer.Write(data);
writer.Flush();
writer.Close();

© Stack Overflow or respective owner

Related posts about c#

Related posts about webclient