Linux - preventing an application from failing due to lack of disk space [migrated]
- by Jernej
Due to an unpredicted scenario I am currently in need of finding a solution to the fact that an application (which I do not wish to kill) is slowly hogging the entire disk space. To give more context
I have an application in Python that uses multiprocessing.Pool to start 5 threads. Each thread writes some data to its own file.
The program is running on Linux and I do not have root access to the machine.
The program is CPU intensive and has been running for months. It still has a few days to write all the data.
40% of the data in the files is redundant and can be removed after a quick test.
The system on which the program is running only has 30GB of remaining disk space and at the current rate of work it will surely be hogged before the program finishes.
Given the above points I see the following solutions with respective problems
Given that the process number i is writing to file_i, is it safe to move file_i to an external location? Will the OS simply create a new instance of file_i and write to it? I assume moving the file would remove it and the process would end up writing to a "dead" file?
Is there a "command line" way to stop 4 of the 5 spawned workers and wait until one of them finishes and then resume their work? (I am sure one single worker thread would avoid hogging the disk)
Suppose I use CTRL+Z to freeze the main process. Will this stop all the other processes spawned by multiprocessing.Pool? If yes, can I then safely edit the files as to remove the redundant lines?
Given the three options that I see, would any of them work in this context? If not, is there a better way to handle this problem? I would really like to avoid the scenario in which the program crashes just few days before its finish.
