What's the most scalable way to handle somewhat large file uploads in a Python webapp?

Posted by Jason Baker on Stack Overflow See other posts from Stack Overflow or by Jason Baker
Published on 2010-05-20T17:17:10Z Indexed on 2010/05/20 17:20 UTC
Read the original article Hit count: 198

Filed under:

python

|

fileupload

|

server

|

asynchronous

We have a web application that takes file uploads for some parts. The file uploads aren't terribly big (mostly word documents and such), but they're much larger than your typical web request and they tend to tie up our threaded servers (zope 2 servers running behind an Apache proxy).

I'm mostly in the brainstorming phase right now and trying to figure out a general technique to use. Some ideas I have are:

Using a python asynchronous server like tornado or diesel or gunicorn.
Writing something in twisted to handle it.
Just using nginx to handle the actual file uploads.

It's surprisingly difficult to find information on which approach I should be taking. I'm sure there are plenty of details that would be needed to make an actual decision, but I'm more worried about figuring out how to make this decision than anything else. Can anyone give me some advice about how to proceed with this?

© Stack Overflow or respective owner

Related posts about python

unmet dependencies in Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried today to install a dvb-card on my Ubuntu 12.04 (Linux blauhai-linux 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ). The installation failed with an error. After that, i tried to install python (it was already installed but i got this error): linux:~$… >>> More
How can I get sikuli-ide to work?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I installed sikuli-ide with sudo apt-get install sikuli-ide Everything was fine until I tried to start it from the terminal. I typed sikuli-ide But the only response I got was [info] locale: en_US The application was not started, furthermore there is no desktop file and sikuli-ide does not… >>> More
Getting PATH right for python after MacPorts install

as seen on Super User - Search for 'Super User'
I can't import some python libraries (PIL, psycopg2) that I just installed with MacPorts. I looked through these forums, and tried to adjust my PATH variable in $HOME/.bash_profile in order to fix this but it did not work. I added the location of PIL and psycopg2 to PATH. I know that Terminal is… >>> More
call python with system() in R to run a python script emulating the python console

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to pass a chunk of Python code to Python in R with something like system('python ...'), and I'm wondering if there is an easy way to emulate the python console in this case. For example, suppose the code is "print 'hello world'", how can I get the output like this in R? >>> print… >>> More
Python - Calling a non python program from python?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am currently struggling to call a non python program from a python script. I have a ~1000 files that when passed through this C++ program will generate ~1000 outputs. Each output file must have a distinct name. The command I wish to run is of the form: program_name -input -output -o1 -o2… >>> More

Related posts about fileupload

fileupload control to show only excel files ?

as seen on Stack Overflow - Search for 'Stack Overflow'
i have a fileupload control in a web page writen in asp & c#.net . on clicking the browse button it shows all the files in the upload dialog,i want to show only files of excel to be shown .how can it be done? else do we have any other tools to show only excel files on clicking browse buttons? >>> More
Apache Commons FileUpload Does not behave like expected

as seen on Stack Overflow - Search for 'Stack Overflow'
I just want to assure you guys that before I post this, I have read couple of similar post but none solved my problem. So here it is. I use Ajax taglib for my form since it wont refresh my screen <%@ taglib uri="WEB-INF/taglib.tld" prefix="a" %> <a:AjaxUpload action="UploadServlet">… >>> More
Getting extension of the file in FileUpload Control

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi At the moment i get file extension of the file like : string fileExt = System.IO.Path.GetExtension(filUpload.FileName); But if the user change the file extension of the file ( for example user could rename "test.txt" to "text.jpg" ), I can't get the real extension . What's the solution ? >>> More
rich:fileupload inside rich:modalpanel doesnt send uploadcomplete event

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm trying to use the rich:fileupload from inside a rich:modalpanel component. The problem is that, when inside modalpanel, fileupload doesn't doesnt start the fileuploadeventlistener method. The same fileupload component works fine outside of modalpanel. (Although it doesn't rerender the components… >>> More
GoogleAppEngine : possible to disable FileUpload?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, When I deploy my application to GoogleAppEngine I keep getting the following error Uncaught exception from servlet java.lang.NoClassDefFoundError: java.io.FileOutputStream is a restricted class. Please see the Google App Engine developer's guide for more details. at com.google.apphosting… >>> More