I'm recently started using monit to monitor the status of sshd on my CentOS 5.4 server. This works fine, but every so often monit reports that sshd is no longer running. This isn't true - I am still able to login to the server via ssh, however I note the following:
There is no longer any PID file at /var/run/sshd.pid - after a reboot this file exists. Once it is gone, restarting sshd via service sshd restart does not create the PID file.
sudo service sshd status reports openssh-daemon is stopped - again, restarting sshd does not change this, but a reboot does.
sudo service sshd stop reports failed, presumably because of the missing PID file.
Any idea what is going on?
Update
sudo netstat -lptun gives the following output relating to port 22
tcp 0 0 :::22 :::* LISTEN 20735/sshd
Killing the process with this PID as suggested by @Henry and then starting sshd via service results in service sshd status recognising the process by PID again. Would still like to understand this better.
RPM verify suggested by a couple of answerers shows this:
sudo rpm -vV openssh openssh-server openssh-clients | grep 'S\.5'
S.5....T c /etc/pam.d/sshd
S.5....T c /etc/ssh/sshd_config
/etc/pam.d/sshd has the following contents:
#%PAM-1.0
auth include system-auth
account required pam_nologin.so
account include system-auth
password include system-auth
session optional pam_keyinit.so force revoke
session include system-auth
#session required pam_loginuid.so
Should that last line be commented out?
Update
Here's the output of @YannickGirouard 's script:
$ sudo ./sshd_test
Searching for the process listening on port 22...
Found the following PID: 21330
Command line for PID 21330: /usr/sbin/sshd
Listing process(es) relating to PID 21330:
UID PID PPID C STIME TTY TIME CMD
root 21330 1 0 14:04 ? 00:00:00 /usr/sbin/sshd
Listing RPM information about openssh packages:
Name : openssh Relocations: (not relocatable)
Version : 4.3p2 Vendor: CentOS
Release : 72.el5_7.5 Build Date: Tue 30 Aug 2011 12:34:14 AM BST
Install Date: Sun 06 Nov 2011 12:50:57 AM GMT Build Host: builder10.centos.org
Group : Applications/Internet Source RPM: openssh-4.3p2-72.el5_7.5.src.rpm
Size : 745390 License: BSD
Signature : DSA/SHA1, Fri 02 Sep 2011 01:13:01 AM BST, Key ID a8a447dce8562897
URL : http://www.openssh.com/portable.html
Summary : The OpenSSH implementation of SSH protocol versions 1 and 2
------------------------------------------------------
Name : openssh-clients Relocations: (not relocatable)
Version : 4.3p2 Vendor: CentOS
Release : 72.el5_7.5 Build Date: Tue 30 Aug 2011 12:34:14 AM BST
Install Date: Sun 06 Nov 2011 12:51:04 AM GMT Build Host: builder10.centos.org
Group : Applications/Internet Source RPM: openssh-4.3p2-72.el5_7.5.src.rpm
Size : 871132 License: BSD
Signature : DSA/SHA1, Fri 02 Sep 2011 01:13:01 AM BST, Key ID a8a447dce8562897
URL : http://www.openssh.com/portable.html
Summary : The OpenSSH client applications
------------------------------------------------------
Name : openssh-server Relocations: (not relocatable)
Version : 4.3p2 Vendor: CentOS
Release : 72.el5_7.5 Build Date: Tue 30 Aug 2011 12:34:14 AM BST
Install Date: Sun 06 Nov 2011 12:51:04 AM GMT Build Host: builder10.centos.org
Group : System Environment/Daemons Source RPM: openssh-4.3p2-72.el5_7.5.src.rpm
Size : 492478 License: BSD
Signature : DSA/SHA1, Fri 02 Sep 2011 01:13:01 AM BST, Key ID a8a447dce8562897
URL : http://www.openssh.com/portable.html
Summary : The OpenSSH server daemon
------------------------------------------------------
However, I've since got things working by killing the process and starting afresh, as suggested by @Henry below, so perhaps I am no longer seeing the same thing. Will try again if I am seeing the issue again after next reboot.
Update - 14 March
Monit alerted me that sshd had disappeared, and again I am able to ssh onto the server. So now I can run the script
$ sudo ./sshd_test
Searching for the process listening on port 22...
Found the following PID: 2208
Command line for PID 2208: /usr/sbin/sshd
Listing process(es) relating to PID 2208:
UID PID PPID C STIME TTY TIME CMD
root 2208 1 0 Mar13 ? 00:00:00 /usr/sbin/sshd
root 1885 2208 0 21:50 ? 00:00:00 sshd: dunx [priv]
Listing RPM information about openssh packages:
Name : openssh Relocations: (not relocatable)
Version : 4.3p2 Vendor: CentOS
Release : 72.el5_7.5 Build Date: Tue 30 Aug 2011 12:34:14 AM BST
Install Date: Sun 06 Nov 2011 12:50:57 AM GMT Build Host: builder10.centos.org
Group : Applications/Internet Source RPM: openssh-4.3p2-72.el5_7.5.src.rpm
Size : 745390 License: BSD
Signature : DSA/SHA1, Fri 02 Sep 2011 01:13:01 AM BST, Key ID a8a447dce8562897
URL : http://www.openssh.com/portable.html
Summary : The OpenSSH implementation of SSH protocol versions 1 and 2
------------------------------------------------------
Name : openssh-clients Relocations: (not relocatable)
Version : 4.3p2 Vendor: CentOS
Release : 72.el5_7.5 Build Date: Tue 30 Aug 2011 12:34:14 AM BST
Install Date: Sun 06 Nov 2011 12:51:04 AM GMT Build Host: builder10.centos.org
Group : Applications/Internet Source RPM: openssh-4.3p2-72.el5_7.5.src.rpm
Size : 871132 License: BSD
Signature : DSA/SHA1, Fri 02 Sep 2011 01:13:01 AM BST, Key ID a8a447dce8562897
URL : http://www.openssh.com/portable.html
Summary : The OpenSSH client applications
------------------------------------------------------
Name : openssh-server Relocations: (not relocatable)
Version : 4.3p2 Vendor: CentOS
Release : 72.el5_7.5 Build Date: Tue 30 Aug 2011 12:34:14 AM BST
Install Date: Sun 06 Nov 2011 12:51:04 AM GMT Build Host: builder10.centos.org
Group : System Environment/Daemons Source RPM: openssh-4.3p2-72.el5_7.5.src.rpm
Size : 492478 License: BSD
Signature : DSA/SHA1, Fri 02 Sep 2011 01:13:01 AM BST, Key ID a8a447dce8562897
URL : http://www.openssh.com/portable.html
Summary : The OpenSSH server daemon
------------------------------------------------------
Again, when I look for /var/run/sshd.pid I don't find it.
$ cat /var/run/sshd.pid
cat: /var/run/sshd.pid: No such file or directory
$ sudo netstat -anp | grep sshd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 2208/sshd
$ sudo kill 2208
$ sudo service sshd start
Starting sshd: [ OK ]
$ cat /var/run/sshd.pid
3794
$ sudo service sshd status
openssh-daemon (pid 3794) is running...
Is it possible that sshd is restarting and not creating a pidfile for some reason?
