I want to scrape a site using GAE and post the results into a Google Entity

Posted by cozza on Stack Overflow See other posts from Stack Overflow or by cozza
Published on 2010-03-09T03:22:41Z Indexed on 2010/03/09 5:51 UTC
Read the original article Hit count: 301

Filed under:

python

|

google-app-engine

|

screen-scraping

I want to scrape this URL : https://www.xstreetsl.com/modules.php?searchSubmitImage_x=0&searchSubmitImage_y=0&SearchLocale=0&name=Marketplace&SearchKeyword=business&searchSubmitImage.x=0&searchSubmitImage.y=0&SearchLocale=0&SearchPriceMin=&SearchPriceMax=&SearchRatingMin=&SearchRatingMax=&sort=&dir=asc

Go into each of the links and extract out various pieces of information e.g. permissions, prims etc then post the results into a Entity on google app engine.

I want to know the best way to go about it?

Chris

© Stack Overflow or respective owner

Related posts about python

unmet dependencies in Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried today to install a dvb-card on my Ubuntu 12.04 (Linux blauhai-linux 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ). The installation failed with an error. After that, i tried to install python (it was already installed but i got this error): linux:~$… >>> More
How can I get sikuli-ide to work?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I installed sikuli-ide with sudo apt-get install sikuli-ide Everything was fine until I tried to start it from the terminal. I typed sikuli-ide But the only response I got was [info] locale: en_US The application was not started, furthermore there is no desktop file and sikuli-ide does not… >>> More
Getting PATH right for python after MacPorts install

as seen on Super User - Search for 'Super User'
I can't import some python libraries (PIL, psycopg2) that I just installed with MacPorts. I looked through these forums, and tried to adjust my PATH variable in $HOME/.bash_profile in order to fix this but it did not work. I added the location of PIL and psycopg2 to PATH. I know that Terminal is… >>> More
call python with system() in R to run a python script emulating the python console

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to pass a chunk of Python code to Python in R with something like system('python ...'), and I'm wondering if there is an easy way to emulate the python console in this case. For example, suppose the code is "print 'hello world'", how can I get the output like this in R? >>> print… >>> More
Python - Calling a non python program from python?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am currently struggling to call a non python program from a python script. I have a ~1000 files that when passed through this C++ program will generate ~1000 outputs. Each output file must have a distinct name. The command I wish to run is of the form: program_name -input -output -o1 -o2… >>> More

Related posts about google-app-engine

unittest import error with virtualenv + google-app-engine-django

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm working with google-app-engine-django + zipped django. Just running "python manage.py test" succeeded without error. But with virtualenv, test was failed with "import unittest error". same error with Django 1.1. - OSX 10.5.6 - google-app-engine-django (r101 via svn) : r100 was failed with launcher… >>> More
Could not import Django settings into Google App Engine

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello all you Google App Engine experts, I have used Django a little before but am new to Google App Engine and am trying to use it's development web server with Django for the first time. I don't know if this is relevent but I previously had Django 1.1 and Python 2.6 on my Windows XP and even… >>> More
How does Google App Engine precompile Java ?

as seen on Stack Overflow - Search for 'Stack Overflow'
App Engine uses a "precompilation" process with the Java bytecode of an app to enhance the performance of the app in the Java runtime environment. Precompiled code functions identically to the original bytecode. Is there any detailed information what this does? >>> More
Creating a session scoped bean in Google App Engine using Spring 2.5

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am trying to create a session bean in spring mvc. I am having the following error message when I run my google app engine server in my local box: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'siteController' defined in ServletContext resource [/WEB-INF/springapp-servlet… >>> More
Google App Engine UI Widgets

as seen on Stack Overflow - Search for 'Stack Overflow'
Are there any UI widgets available to the python side of Google App Engine? I'd like something like the collapsed/expanded views of Google Groups threads. Are these type things limited to the GWT side? >>> More