Using Scrapy Spider in a CGI Script or Web Framework, ie: Django

Posted by Israel ANY on Stack Overflow See other posts from Stack Overflow or by Israel ANY
Published on 2010-05-24T22:37:15Z Indexed on 2010/05/24 22:41 UTC
Read the original article Hit count: 375

Filed under:

python

|

django

|

cgi

|

scraping

|

scrapy

How do I use a Python library such as Scrapy in a CGI script or even in a framework such as Django if I need to do so later?

Here is the documentation I've consulted thus far, but it doesn't seem to meet the concern I have.

http://doc.scrapy.org/topics/spiders.html
http://doc.scrapy.org/topics/webconsole.html

Critique and suggestions are welcomed!

© Stack Overflow or respective owner

Related posts about python

unmet dependencies in Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried today to install a dvb-card on my Ubuntu 12.04 (Linux blauhai-linux 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ). The installation failed with an error. After that, i tried to install python (it was already installed but i got this error): linux:~$… >>> More
How can I get sikuli-ide to work?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I installed sikuli-ide with sudo apt-get install sikuli-ide Everything was fine until I tried to start it from the terminal. I typed sikuli-ide But the only response I got was [info] locale: en_US The application was not started, furthermore there is no desktop file and sikuli-ide does not… >>> More
Getting PATH right for python after MacPorts install

as seen on Super User - Search for 'Super User'
I can't import some python libraries (PIL, psycopg2) that I just installed with MacPorts. I looked through these forums, and tried to adjust my PATH variable in $HOME/.bash_profile in order to fix this but it did not work. I added the location of PIL and psycopg2 to PATH. I know that Terminal is… >>> More
call python with system() in R to run a python script emulating the python console

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to pass a chunk of Python code to Python in R with something like system('python ...'), and I'm wondering if there is an easy way to emulate the python console in this case. For example, suppose the code is "print 'hello world'", how can I get the output like this in R? >>> print… >>> More
Python - Calling a non python program from python?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am currently struggling to call a non python program from a python script. I have a ~1000 files that when passed through this C++ program will generate ~1000 outputs. Each output file must have a distinct name. The command I wish to run is of the form: program_name -input -output -o1 -o2… >>> More

Related posts about django

Overriding the save() method of a model that uses django-mptt

as seen on Stack Overflow - Search for 'Stack Overflow'
I've been using django-mptt in my project for a while now, it's fabulous. Recently, I've found a need to override a model's save() method that uses mptt, and I'm getting an error when I try to save a new instance of that model: Exception Type: ValueError at /admin/scrivener/page/add/ Exception Value:… >>> More
Where is meta.local_fields set in django.db.models.base.py ?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm getting the error: Exception Value: (1110, "Column 'about' specified twice") As I was reviewing the Django error page, I noticed that the customizations the model User, seem to be appended to the List twice. This seems to be happening here in django/db/model/base.py in base_save(): values… >>> More
Deploying Django on EC2 using Bitnami Djangostack: WSGI script cannot be loadded

as seen on Stack Overflow - Search for 'Stack Overflow'
I've been struggling to deploy Django application on Amazon EC2 using Bitnami Djangostack for the last couple of days. When I go to http://dewey.io I see the default bitnami page (/opt/bitnami/apache2/htdocs/index.html), however, when I open http://dewey.io/portnoy, I get 'Internal Server Error'… >>> More
Internal Server Error with mod_wsgi [django] on windows xp

as seen on Stack Overflow - Search for 'Stack Overflow'
when i run development server it works very well, even an empty project runing in mod_wsgi i have no problem but when i want to put my own project i get an Internal Server Error (500) in my apache conf i put WSGIScriptAlias /codevents C:/django/apache/CODEvents.wsgi <Directory "C:/django/apache"> Order… >>> More
Internal Server Error with mod_wgsi [django] on windows xp

as seen on Stack Overflow - Search for 'Stack Overflow'
when i run development server it works very well, even an empty project runing in mod_wsgi i have no problem but when i want to put my own project i get an Internal Server Error (500) in my apache conf i put WSGIScriptAlias /codevents C:/django/apache/CODEvents.wsgi <Directory "C:/django/apache"> Order… >>> More