Python: Huge file reading by using linecache Vs normal file access open()

Posted by user335223 on Stack Overflow See other posts from Stack Overflow or by user335223
Published on 2010-05-07T08:45:36Z Indexed on 2010/05/07 8:58 UTC
Read the original article Hit count: 200

Filed under:

python

|

threads

|

reading

|

huge

|

file

Hi, I am in a situation where multiple threads reading the same huge file with mutliple file pointers to same file. The file will have atleast 1 million lines. Eachline's length varies from 500 characters to 1500 characters. There won't "write" operations on the file. Each thread will start reading the same file from different lines. Which is the efficient way..? Using the Python's linecache or normal readline() or is there anyother effient way?

© Stack Overflow or respective owner

Related posts about python

unmet dependencies in Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried today to install a dvb-card on my Ubuntu 12.04 (Linux blauhai-linux 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ). The installation failed with an error. After that, i tried to install python (it was already installed but i got this error): linux:~$… >>> More
How can I get sikuli-ide to work?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I installed sikuli-ide with sudo apt-get install sikuli-ide Everything was fine until I tried to start it from the terminal. I typed sikuli-ide But the only response I got was [info] locale: en_US The application was not started, furthermore there is no desktop file and sikuli-ide does not… >>> More
Getting PATH right for python after MacPorts install

as seen on Super User - Search for 'Super User'
I can't import some python libraries (PIL, psycopg2) that I just installed with MacPorts. I looked through these forums, and tried to adjust my PATH variable in $HOME/.bash_profile in order to fix this but it did not work. I added the location of PIL and psycopg2 to PATH. I know that Terminal is… >>> More
call python with system() in R to run a python script emulating the python console

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to pass a chunk of Python code to Python in R with something like system('python ...'), and I'm wondering if there is an easy way to emulate the python console in this case. For example, suppose the code is "print 'hello world'", how can I get the output like this in R? >>> print… >>> More
Python - Calling a non python program from python?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am currently struggling to call a non python program from a python script. I have a ~1000 files that when passed through this C++ program will generate ~1000 outputs. Each output file must have a distinct name. The command I wish to run is of the form: program_name -input -output -o1 -o2… >>> More

Related posts about threads

Application threads vs Service threads

as seen on Stack Overflow - Search for 'Stack Overflow'
What are the advantages/disadvantages in placing a lengthy network access code in a thread in an activity or a thread in a service? How would it affect the application? I am writing a streaming audio player and from what I've read so far putting the code in a service will still end up blocking the… >>> More
[newbie]Webservice - Threads - close if no active threads, if any activen, then wait till its compl

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi I have an ASP.Net webservice running. When the system date changes, i want to close all the threads and restart them again, if the thread/s aren't doing any work. If any of the thread/s is in the process of doing some work(active threads), then i need to wait till the current thread/s complete… >>> More
Multiple threads or process with threads

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, this is for an assignment so I'm not looking for code. I have to simulate a game where each player has turns and needs to 'pay attention' to what's going on. So far, i know I'll need two threads for each player, one that will sleep until the player's turn and the other paying attention. My… >>> More
Tomcat threads vs Java threads

as seen on Stack Overflow - Search for 'Stack Overflow'
When using java threads, one has to take care of the basic problems that come with concurrency through synchronization etc. AFAIK Tomcat also works with threads to handle its workload. Why is it, that I don't have to think about making my code threadsafe when it is running in Tomcat? >>> More
C# threads - Posting messages between threads

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi All, I am working on a project which involves reusing as well as migrating some of the existing MFC code to C#. The current code in MFC, creates some threads and uses ::PostthreadeMessage() and ON_THREAD_MESSAGE(msg,func) for inter thread asynchronous communication through messages. ::PostthreadeMessage()… >>> More