HttpWebRequest does has empty response requesting a search from Bing

Posted by Jarrod Maxwell on Stack Overflow See other posts from Stack Overflow or by Jarrod Maxwell
Published on 2010-04-07T11:56:53Z Indexed on 2010/04/07 12:23 UTC
Read the original article Hit count: 496

Filed under:
|
|
|
|

I have the following code that sends a HttpWebRequest to Bing. When I request the url below though it returns what appears to be an empty response when it should be returning a list of results.

            var response = string.Empty;
            var httpWebRequest = WebRequest.Create("http://www.bing.com/search?q=stackoverflow&count=100") as HttpWebRequest;

            httpWebRequest.Method = WebRequestMethods.Http.Get;
            httpWebRequest.Headers.Add("Accept-Language", "en-US");
            httpWebRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Win32)";
            httpWebRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");

            using (var httpWebResponse = httpWebRequest.GetResponse() as HttpWebResponse)
            {
                Stream stream = null;
                using (stream = httpWebResponse.GetResponseStream())
                {
                    if (httpWebResponse.ContentEncoding.ToLower().Contains("gzip"))
                        stream = new GZipStream(stream, CompressionMode.Decompress);
                    else if (httpWebResponse.ContentEncoding.ToLower().Contains("deflate"))
                        stream = new DeflateStream(stream, CompressionMode.Decompress);

                    var streamReader = new StreamReader(stream, Encoding.UTF8);
                    response = streamReader.ReadToEnd();
                }
            }

Its pretty standard code for requesting and receiving a web page. Any ideas why the response is empty? Thanks in advance.

EDIT I left off a query string parameter in the url. I also had &count=100 which I have now corrected. It seems to work for values of 50 and below but returns nothing when larger. This works ok when in the browser, but not for this web request.

It makes me think the issue is that the response is large and HttpWebResponse is not handling that for me the way I have it set up. Just a guess though.

© Stack Overflow or respective owner

Related posts about httpwebrequest

Related posts about c#