Get Zipped Logs from a Remote Server

Posted by Jonathan on Server Fault See other posts from Server Fault or by Jonathan
Published on 2012-10-17T21:26:39Z Indexed on 2012/10/17 23:03 UTC
Read the original article Hit count: 200

Filed under:

log-files

|

scp

|

rysnc

I am tasked with trying to find a way to download zipped logs from a remote server. There are quite a bit of these logs and they are constantly created.

I do have limited ssh access to the remote server and can scp or rsync the files.

However, due to the sheer size of these logs file, I do not want to rsync all of them. The logs could get to terabytes and for rsync to compare them may take some time. I only want to get any new file that was created/last updated an hour ago.

I also am worried that I will rsync logs that are in the process of being created, so I was thinking to only rsync files that were last modified 3-5 minutes ago.

Would anyone be so kind as to help me with such a process? Thank you in advance.

© Server Fault or respective owner

Related posts about log-files

Problem with squid log files

as seen on Server Fault - Search for 'Server Fault'
I am using SARG to get a report on the squid log files, I get this result /usr/local/Sarg/bin/sarg -l /usr/local/squid/var/logs/access.log SARG: Records in file: 0, reading: 0.00% SARG: Records in file: 0, reading: 0.00% SARG: Records in file: 0, reading: 0.00% SARG: Records in file: 0, reading:… >>> More
Safe to enable compression C:\WINDOWS\system32\LogFiles

as seen on Server Fault - Search for 'Server Fault'
Will it slow down my website if I enable compression on the C:\WINDOWS\system32\LogFiles directory? These files are slowly but surely chewing away drive space and I'd like to slow the expansion. I'm worried however that this will also slow my websites. >>> More
My linux server time and log files are not the same

as seen on Server Fault - Search for 'Server Fault'
Hi i have installed NTP on my linux server and i am getting my clock from a 6500 core switch, everything is working fine. When i ssh to a switch i have it all sent to a log file on the linux server, this log file does not time stamp with the same time as i have on the server. the date and hwclock… >>> More
Minimizing SQL transaction log file size on developer box running simple recovery model

as seen on Server Fault - Search for 'Server Fault'
We have alot of SQL servers on development environment where we never take backup of the databases (TFS for code is enough). The (SharePoint) databases are all set to simple recovery model, but the log files, especially for the SharePoint configuration database is growing quite large and filling… >>> More
How to write to Tomcat log files

as seen on Stack Overflow - Search for 'Stack Overflow'
What is the easiest way to print out something in a Tomcat Log file. I was under the impression that System.out.println() would print to one of the log files. I just want to print out the value of a string, so the simplest solution would be best. If there is no simple/trivial solution I guess I… >>> More

Related posts about scp

Variable directory names over SCP

as seen on Server Fault - Search for 'Server Fault'
We have a backup routine that previously ran from one disk to another on the same server, but have recently moved the source data to a remote server and are trying to replicate the job via scp. We need to run the script on the target server, and we've set up key-based scp (no username/password… >>> More
Does SCP lock the file it is transfering?

as seen on Server Fault - Search for 'Server Fault'
Have a situation where we have an application log sitting out on an AIX server. The log is being continually written to from the application, and we have users on Windows who want to view the file. What they've been doing is using WinSCP to transfer the file to their desktop and they open it using… >>> More
Problems with SCP stalling during file copy over VPN

as seen on Server Fault - Search for 'Server Fault'
I have a series of files I need to copy via SCP over a VPN to a remote linux server each night. The files are not large, we're talking about tens of megabytes here, but the file copy almost always stalls after a few seconds. Running the SCP command with -vvv, I see the following over and over throughout… >>> More
scp hangs at 100%, before finishing

as seen on Super User - Search for 'Super User'
Hi, my scp commands do hang at 100% before the finish (some seconds), The output I get using -vvv is this: http://paste.pocoo.org/show/137504/ Any ideas? >>> More
Prevent scp from copying local files?

as seen on Super User - Search for 'Super User'
I just read the Linux scp command issue question and it reminded me that I regularily forget to specify the colon in the host part of a scp command, and thus copying a file locally instead of copying to a remote host, e.g. I do scp foo host instead of scp foo host: But I never use scp to copy… >>> More