Help converting code using httlib2 to use urllib2

Posted by ThinkCode on Stack Overflow See other posts from Stack Overflow or by ThinkCode
Published on 2010-06-02T18:43:35Z Indexed on 2010/06/02 19:04 UTC
Read the original article Hit count: 278

What am I trying to do?

Visit a site, retrieve cookie, visit the next page by sending in the cookie info. It all works but httplib2 is giving me one too many problems with socks proxy on one site.

http = httplib2.Http()
main_url = 'http://mywebsite.com/get.aspx?id='+ id +'&rows=25'
response, content = http.request(main_url, 'GET', headers=headers)
main_cookie = response['set-cookie']
referer = 'http://google.com'
headers = {'Content-type': 'application/x-www-form-urlencoded', 'Cookie': main_cookie, 'User-Agent' : USER_AGENT, 'Referer' : referer}

How to do the same exact thing using urllib2 (cookie retrieving, passing to the next page on the same site)?

Thank you.

© Stack Overflow or respective owner

Related posts about python

Related posts about screen-scraping