How do you monitor and react when some scheduled job fails? - general question

Posted by Dzida on Server Fault See other posts from Server Fault or by Dzida
Published on 2010-06-10T09:42:24Z Indexed on 2010/06/10 9:53 UTC
Read the original article Hit count: 168

Filed under:

monitoring

|

cron

|

scheduled-task

|

failure

|

silent

Hi,

In many projects my team faced problems with 'silent fails' of some important components. There are lot of tasks executed behind the scenes and if somethings fails (either by errors in logic or hardware problems) in most cases responsible person is not notified (or not notified instantly).

I know about heavy-weight monitoring tools that could solve some of that problems but there over-complicated and too expensive for our team.

I am interested what are your solutions for such problems.

© Server Fault or respective owner

Related posts about monitoring

Issues with signal handling [closed]

as seen on Programmers - Search for 'Programmers'
I am trying to actually study the signal handling behavior in multiprocess system. I have a system where there are three signal generating processes generating signals of type SIGUSR1 and SIGUSR1. I have two handler processes that handle a particular type of signal. I have another monitoring process… >>> More
JMX Based Monitoring - Part Two - JVM Monitoring

as seen on Oracle Blogs - Search for 'Oracle Blogs'
This the second article in the series focussing on the JMX based monitoring capabilities possible with the Oracle Utilities Application Framework. In all versions of the Oracle utilities Application Framework, it is possible to use the basic JMX based monitoring available with the Java Virtual Machine… >>> More
JMX Based Monitoring - Part Three - Web App Server Monitoring

as seen on Oracle Blogs - Search for 'Oracle Blogs'
In the last blog entry I showed a technique for integrating a JMX console with Oracle WebLogic which is a standard feature of Oracle WebLogic 11g. Customers on other Web Application servers and other versions of Oracle WebLogic can refer to the documentation provided with the server to do a similar… >>> More
JMX Based Monitoring - Part Four - Business App Server Monitoring

as seen on Oracle Blogs - Search for 'Oracle Blogs'
In the last blog entry I talked about the Oracle Utilities Application Framework V4 feature for monitoring and managing aspects of the Web Application Server using JMX. In this blog entry I am going to discuss a similar new feature that allows JMX to be used for management and monitoring the Oracle… >>> More
Keeping track of File System Utilization in Ops Center 12c

as seen on Oracle Blogs - Search for 'Oracle Blogs'
Enterprise Manager Ops Center 12c provides significant monitoring capabilities, combined with very flexible incident management. These capabilities even extend to monitoring the file systems associated with Solaris or Linux assets. Depending on your needs you can monitor and manage incidents, or you… >>> More

Related posts about cron

any clue in these logs why keyboard audio and internet are messed up

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
Jun 7 00:01:18 Isis lightdm: pam_unix(lightdm-autologin:session): session opened for user mimi by (uid=0) Jun 7 00:01:18 Isis lightdm: pam_ck_connector(lightdm-autologin:session): nox11 mode, ignoring PAM_TTY :0 Jun 7 00:01:26 Isis polkitd(authority=local): Registered Authentication Agent for… >>> More
cron.daily not running at the time it should?

as seen on Server Fault - Search for 'Server Fault'
My /etc/cron.daily scripts seem to be executing far later from what I understand they should. I am in Ubuntu and anacron is installed. If I do a sudo cat /var/log/syslog | grep cron I get something like: Aug 23 01:17:01 mymachine CRON[25171]: (root) CMD ( cd / && run-parts --report /etc/cron… >>> More
mdadm cron job sends email that cron has run

as seen on Server Fault - Search for 'Server Fault'
I've got an Ubuntu 8.04 server using mdadm to create several RAID1 arrays. I created /etc/cron.hourly/mdadm as follows: #! /bin/sh set -e mdadm --monitor /dev/md0 /dev/md3 /dev/md4 --oneshot (Yes, the array numbers are not sequential, and I'm not using --scan beacuse I have a degraded array that… >>> More
Cron Tips for not running cron jobs on holidays (the monday of a three day weekend)

as seen on Super User - Search for 'Super User'
We have about a one hundred machine set-up with each machine running cron jobs like starting and stopping services and archiving these services' log files at the end of the day to a centralized repository. One headache we have is the three-day weekend (we're closed on holidays). We don't want the… >>> More
cron doesn't execute it's commands

as seen on Server Fault - Search for 'Server Fault'
I created an own small server with Debian. Last night i updated it. It created an error while generating the initrd and it didn't boot. Today i booted from another filesystem and did dpkg --configure -a with chroot. I also checked the filesystem. Now everything should be ok. But cron doesn't work:-(… >>> More