How can I remove old log entries from a log file and archive them somewhere else in Linux?
- by Mike B
CentOS 4.x
I apologize in advance if this is not the appropriate place to ask this question. It pertains to a linux server / IT admin task.
I've got a log file on an old CentOS 4.x server and I want to remove log entries older than a certain date and place them in a new file for archive.
Here's an example of the log format:
2012-06-07 22:32:01,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|blah blah blah
2012-06-07 22:32:03,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|blah blah blah
2012-06-07 22:32:04,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|
2012-06-07 22:32:10,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|blah blah blah
2012-06-07 22:32:12,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|blah blah blah
2012-06-07 22:32:15,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|
2012-06-07 22:32:40,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|blah blah blah
2012-06-07 22:32:58,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|blah blah blah
2012-06-07 22:33:01,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|
2012-06-07 22:33:01,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|blah blah blah
2012-06-07 22:33:02,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|
Essentially, I'm looking for a one-liner that will do the following:
Find any events older than a provided YYYY-MM-DD and remove them
from the primary log file.
Take the deleted events from step 1 and put
them in a new log file
(Optional) Compress the new archive log file holding the deleted events.
I'm aware that there are log rotate tools that do this but this should just be a one-time task so I'd prefer not to set that up.
Additional notes:
If the date part it tricky or too resource intensive, an alternative would be to just keep the last X number of lines and move the rest. I was originally thinking of something like tail -n 10000 > newfile.txt but that would mean moving the "good" logs to a new file and then doing a name swap... and then I'd still need to remove the "good" entries from the archive.
This particular log file is pretty large (1 GB) so I'd prefer the task to be as resource and time efficient as possible.
The extra pipes in the log concern me and I'm not sure if I'd need extra protection in the commands to avoid that from causing problems.
