So.. I've installed Logstash, and instead of using the logstash shipper (because it needs the JVM and is generally massive), I'm using rsyslogd with the following configuration.
# Use traditional timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
# Provides kernel logging support (previously done by rklogd)
$ModLoad imklog
# Provides support for local system logging (e.g. via logger command)
$ModLoad imuxsock
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none;local6.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
In /etc/rsyslog.d/logstash.conf there are 28 file monitor blocks using imfile
$ModLoad imfile # Load the imfile input module
$ModLoad imklog # for reading kernel log messages
$ModLoad imuxsock # for reading local syslog messages
$InputFileName /var/log/rabbitmq/startup_err
$InputFileTag rmq-err:
$InputFileStateFile state-rmq-err
$InputFileFacility local6
$InputRunFileMonitor
....
$InputFileName /var/log/some.other.custom.log
$InputFileTag cust-log:
$InputFileStateFile state-cust-log
$InputFileFacility local6
$InputRunFileMonitor
....
*.* @@10.90.0.110:5514
There are 28 InputFileMonitor blocks, each monitoring a different custom application logfile..
If I run
[root@secret-gm02 ~]# lsof|grep rsyslog
rsyslogd 5380 root cwd DIR 253,0 4096 2 /
rsyslogd 5380 root rtd DIR 253,0 4096 2 /
rsyslogd 5380 root txt REG 253,0 278976 1015955 /sbin/rsyslogd
rsyslogd 5380 root mem REG 253,0 58400 1868123 /lib64/libgcc_s-4.1.2-20080825.so.1
rsyslogd 5380 root mem REG 253,0 144776 1867778 /lib64/ld-2.5.so
rsyslogd 5380 root mem REG 253,0 1718232 1867780 /lib64/libc-2.5.so
rsyslogd 5380 root mem REG 253,0 23360 1867787 /lib64/libdl-2.5.so
rsyslogd 5380 root mem REG 253,0 145872 1867797 /lib64/libpthread-2.5.so
rsyslogd 5380 root mem REG 253,0 85544 1867815 /lib64/libz.so.1.2.3
rsyslogd 5380 root mem REG 253,0 53448 1867801 /lib64/librt-2.5.so
rsyslogd 5380 root mem REG 253,0 92816 1868016 /lib64/libresolv-2.5.so
rsyslogd 5380 root mem REG 253,0 20384 1867990 /lib64/rsyslog/lmnsd_ptcp.so
rsyslogd 5380 root mem REG 253,0 53880 1867802 /lib64/libnss_files-2.5.so
rsyslogd 5380 root mem REG 253,0 23736 1867800 /lib64/libnss_dns-2.5.so
rsyslogd 5380 root mem REG 253,0 20768 1867988 /lib64/rsyslog/lmnet.so
rsyslogd 5380 root mem REG 253,0 11488 1867982 /lib64/rsyslog/imfile.so
rsyslogd 5380 root mem REG 253,0 24040 1867983 /lib64/rsyslog/imklog.so
rsyslogd 5380 root mem REG 253,0 11536 1867987 /lib64/rsyslog/imuxsock.so
rsyslogd 5380 root mem REG 253,0 13152 1867989 /lib64/rsyslog/lmnetstrms.so
rsyslogd 5380 root mem REG 253,0 8400 1867992 /lib64/rsyslog/lmtcpclt.so
rsyslogd 5380 root 0r REG 0,3 0 4026531848 /proc/kmsg
rsyslogd 5380 root 1u IPv4 1200589517 0t0 TCP 10.10.10.90 t:40629->10.10.10.90:5514 (ESTABLISHED)
rsyslogd 5380 root 2u IPv4 1200589527 0t0 UDP *:45801
rsyslogd 5380 root 3w REG 253,3 17999744 2621483 /var/log/messages
rsyslogd 5380 root 4w REG 253,3 13383 2621484 /var/log/secure
rsyslogd 5380 root 5w REG 253,3 7180 2621493 /var/log/maillog
rsyslogd 5380 root 6w REG 253,3 43321 2621529 /var/log/cron
rsyslogd 5380 root 7w REG 253,3 0 2621494 /var/log/spooler
rsyslogd 5380 root 8w REG 253,3 0 2621495 /var/log/boot.log
rsyslogd 5380 root 9r REG 253,3 1064271998 2621464 /var/log/custom-application.monolog.log
rsyslogd 5380 root 10u unix 0xffff81081fad2e40 0t0 1200589511 /dev/log
You can see that there are nowhere near 28 logfiles actually being read.
I really had to get one file monitored, so I moved it to the top, and it picked it up, but I'd like to be able to monitor all 28+ files, and not have to worry.
OS is
Centos 5.5
Kernel 2.6.18-308.el5
rsyslogd 3.22.1, compiled with:
FEATURE_REGEXP: Yes
FEATURE_LARGEFILE: Yes
FEATURE_NETZIP (message compression): Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
Atomic operations supported: Yes
Runtime Instrumentation (slow code): No
Questions:
Why is rsyslogd only monitoring a very small subset of the files? How can I fix this so that all the files are monitored?
