Weird vps server issue

Posted by anon-user0 on Server Fault See other posts from Server Fault or by anon-user0
Published on 2012-06-27T10:53:55Z Indexed on 2012/06/27 15:18 UTC
Read the original article Hit count: 658

Filed under:
|
|

I have an unmanaged linux vps Ubuntu 11.10 (Oneiric Ocelot). I have LNMP installed. Also php-fpm php-apc, varnish, memcache.

I have (or rather had) several live sites on it.

under normal load the server uses ~700 mb memory. But since last night its using only 20mb~ memory and a lot of the services seems to be down (according to htop) I only see nginx working and mysql starts up and goes does every few minutes on a loop.

Here are some information on the server that might help you help me:

[email protected]:~# uname  -a
Linux server 2.6.18-308.el5.028stab099.3 #1 SMP Wed Mar 7 15:56:00 MSK 2012 i686 i686 i386 GNU/Linux

-

[email protected]:~# ifconfig -a
lo        Link encap:Local Loopback  
          LOOPBACK  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

venet0    Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:127.0.0.2  P-t-P:127.0.0.2  Bcast:0.0.0.0  Mask:255.255.255.255
          UP BROADCAST POINTOPOINT RUNNING NOARP  MTU:1500  Metric:1
          RX packets:12515 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9541 errors:0 dropped:1 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:7191214 (7.1 MB)  TX bytes:536726 (536.7 KB)

venet0:0  Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:176.31.158.78  P-t-P:176.31.158.78  Bcast:0.0.0.0  Mask:255.255.255.255
          UP BROADCAST POINTOPOINT RUNNING NOARP  MTU:1500  Metric:1

-

[email protected]:~# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 *:http-alt              *:*                     LISTEN     
tcp        0      0 *:ssh                   *:*                     LISTEN     
tcp6       0      0 [::]:http-alt           [::]:*                  LISTEN     
tcp6       0      0 [::]:ssh                [::]:*                  LISTEN     
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ACC ]     STREAM     LISTENING     9307368  @/com/ubuntu/upstart
    -

htop: http://i.stack.imgur.com/NHKYX.png

EDIT: Stressed. mind was not working adding log:

[email protected]:~#  less /var/log/syslog

Jun 27 05:27:42 server syslogd 1.5.0#6ubuntu1: restart.
Jun 27 05:39:01 server CRON[9298]: (root) CMD ([ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) -delete)
Jun 27 05:40:01 server CRON[9463]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jun 27 05:46:21 server sm-msp-queue[9480]: q5R1R7Ue004056: to=root, ctladdr=root (0/0), delay=00:19:14, xdelay=00:06:18, mailer=relay, pri=122407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred:     Connection timed out with [127.0.0.1]
Jun 27 05:52:39 server sm-msp-queue[9480]: q5QMk7S9009582: to=root, ctladdr=root (0/0), delay=03:06:32, xdelay=00:06:18, mailer=relay, pri=842407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 06:00:01 server CRON[15671]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jun 27 06:06:22 server sm-msp-queue[15690]: q5R1R7Ue004056: to=root, ctladdr=root (0/0), delay=00:39:15, xdelay=00:06:18, mailer=relay, pri=212407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 06:09:01 server CRON[18114]: (root) CMD ([ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) -delete)
Jun 27 06:12:40 server sm-msp-queue[15690]: q5QMk7S9009582: to=root, ctladdr=root (0/0), delay=03:26:33, xdelay=00:06:18, mailer=relay, pri=932407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]

Jun 27 06:20:02 server CRON[21888]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)

Jun 27 06:26:22 server sm-msp-queue[21907]: q5R1R7Ue004056: to=root, ctladdr=root (0/0), delay=00:59:15, xdelay=00:06:18, mailer=relay, pri=302407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]

Jun 27 06:27:02 server CRON[24021]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jun 27 06:32:40 server sm-msp-queue[21907]: q5QMk7S9009582: to=root, ctladdr=root (0/0), delay=03:46:33, xdelay=00:06:18, mailer=relay, pri=1022407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 06:39:01 server CRON[27941]: (root) CMD ([ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) -delete)
Jun 27 06:40:02 server CRON[28110]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jun 27 06:46:22 server sm-msp-queue[28125]: q5R1R7Ue004056: to=root, ctladdr=root (0/0), delay=01:19:15, xdelay=00:06:18, mailer=relay, pri=392407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 06:52:40 server sm-msp-queue[28125]: q5QMk7S9009582: to=root, ctladdr=root (0/0), delay=04:06:33, xdelay=00:06:18, mailer=relay, pri=1112407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 06:52:40 server sm-msp-queue[28125]: q5QMk7S9009582: q5R2e4uo028125: sender notify: Warning: could not send message for past 4 hours
Jun 27 06:52:44 server sm-msp-queue[28125]: q5R2e4uo028125: to=root, delay=00:00:04, xdelay=00:00:04, mailer=relay, pri=33690, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 07:00:02 server CRON[1543]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jun 27 07:06:21 server sm-msp-queue[1560]: q5R2e4uo028125: to=root, delay=00:13:41, xdelay=00:06:18, mailer=relay, pri=123690, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 07:09:01 server CRON[3986]: (root) CMD ([ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) -delete)
Jun 27 07:12:39 server sm-msp-queue[1560]: q5R1R7Ue004056: to=root, ctladdr=root (0/0), delay=01:45:32, xdelay=00:06:18, mailer=relay, pri=482407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 07:18:57 server sm-msp-queue[1560]: q5QMk7S9009582: to=root, ctladdr=root (0/0), delay=04:32:50, xdelay=00:06:18, mailer=relay, pri=1202407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 07:20:02 server CRON[7760]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jun 27 07:26:22 server sm-msp-queue[7775]: q5R2e4uo028125: to=root, delay=00:33:42, xdelay=00:06:18, mailer=relay, pri=213690, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 07:27:01 server CRON[9887]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jun 27 07:32:40 server sm-msp-queue[7775]: q5R1R7Ue004056: to=root, ctladdr=root (0/0), delay=02:05:33, xdelay=00:06:18, mailer=relay, pri=572407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 07:38:58 server sm-msp-queue[7775]: q5QMk7S9009582: to=root, ctladdr=root (0/0), delay=04:52:51, xdelay=00:06:18, mailer=relay, pri=1292407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 07:39:01 server CRON[13813]: (root) CMD ([ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth :
  • [email protected]:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/simfs 20G 2.3G 18G 12% /

-

Jun 26 16:22:41 server varnishd[1413]: Child (32425) died signal=3
Jun 26 16:22:41 server varnishd[1413]: child (21687) Started
Jun 26 16:22:41 server varnishd[1413]: Child (21687) said Child starts
Jun 26 16:22:41 server varnishd[1413]: Child (21687) said SMF.s0 mmap'ed 1073741824 bytes of 1073741824
Jun 26 16:34:28 server -- MARK --
Jun 26 16:54:29 server -- MARK --
Jun 26 17:14:29 server -- MARK --
Jun 26 17:34:29 server -- MARK --
Jun 26 17:54:29 server -- MARK --
Jun 26 18:14:29 server -- MARK --
Jun 26 18:34:29 server -- MARK --
Jun 26 18:54:29 server -- MARK --
Jun 26 19:14:29 server -- MARK --
Jun 26 19:34:29 server -- MARK --
Jun 26 19:54:29 server -- MARK --
Jun 26 20:14:29 server -- MARK --
Jun 26 20:34:29 server -- MARK --
Jun 26 20:48:12 server exiting on signal 15
Jun 26 20:51:58 server syslogd 1.5.0#6ubuntu1: restart.
Jun 26 20:52:01 server varnishd[1324]: Platform: Linux,2.6.18-308.el5.028stab099.3,i686,-sfile,-smalloc,-hcritbit 
Jun 26 21:11:58 server -- MARK --
Jun 26 21:31:58 server -- MARK --
Jun 26 21:51:58 server -- MARK --
Jun 26 22:11:58 server -- MARK --
Jun 26 22:31:58 server -- MARK --
Jun 26 22:51:58 server -- MARK --
Jun 26 23:11:58 server -- MARK --
Jun 26 23:31:58 server -- MARK --
Jun 26 23:51:58 server -- MARK --
Jun 27 00:11:58 server -- MARK --
Jun 27 00:23:42 server exiting on signal 15
Jun 27 02:21:10 server syslogd 1.5.0#6ubuntu1: restart.
Jun 27 02:21:12 server varnishd[1341]: Platform: Linux,2.6.18-308.el5.028stab099.3,i686,-sfile,-smalloc,-hcritbit 
Jun 27 02:41:10 server -- MARK --
Jun 27 02:46:41 server syslogd 1.5.0#6ubuntu1: restart.
Jun 27 03:20:44 server syslogd 1.5.0#6ubuntu1: restart.
Jun 27 03:20:46 server varnishd[1238]: Platform: Linux,2.6.18-308.el5.028stab099.3,i686,-sfile,-smalloc,-hcritbit 
Jun 27 03:20:46 server varnishd[1238]: child (1239) Started
Jun 27 03:20:46 server varnishd[1238]: Child (1239) said Child starts
Jun 27 03:20:46 server varnishd[1238]: Child (1239) said SMF.s0 mmap'ed 1073741824 bytes of 1073741824
Jun 27 03:32:52 server exiting on signal 15
Jun 27 03:33:16 server syslogd 1.5.0#6ubuntu1: restart.
Jun 27 03:33:31 server varnishd[1372]: Platform: Linux,2.6.18-308.el5.028stab099.3,i686,-sfile,-smalloc,-hcritbit 
Jun 27 03:53:16 server -- MARK --
Jun 27 04:13:16 server -- MARK --
Jun 27 04:33:16 server -- MARK --
Jun 27 04:53:16 server -- MARK --
Jun 27 05:13:16 server -- MARK --
Jun 27 05:27:42 server syslogd 1.5.0#6ubuntu1: restart.
Jun 27 05:53:17 server -- MARK --
Jun 27 06:13:17 server -- MARK --
Jun 27 06:33:17 server -- MARK --
Jun 27 06:53:17 server -- MARK --
Jun 27 07:13:17 server -- MARK --
Jun 27 07:33:17 server -- MARK --
Jun 27 07:53:17 server -- MARK --
Jun 27 08:13:17 server -- MARK --
Jun 27 08:33:17 server -- MARK --
Jun 27 08:53:17 server -- MARK --
Jun 27 09:13:17 server -- MARK --
Jun 27 09:33:17 server -- MARK --
Jun 27 09:53:17 server -- MARK --
Jun 27 10:13:17 server -- MARK --
Jun 27 10:33:17 server -- MARK --
Jun 27 10:53:17 server -- MARK --
Jun 27 11:13:17 server -- MARK --
Jun 27 11:33:17 server -- MARK --
Jun 27 11:53:18 server -- MARK --
Jun 27 12:13:18 server -- MARK --
Jun 27 12:33:18 server -- MARK --
Jun 27 12:53:18 server -- MARK --
Jun 27 13:13:18 server -- MARK --
Jun 27 13:33:18 server -- MARK --
Jun 27 13:53:18 server -- MARK --
Jun 27 14:13:18 server -- MARK --
Jun 27 14:33:18 server -- MARK --
Jun 27 14:53:18 server -- MARK --

--

[email protected]:~# cat /var/log/nginx/error.log
2012/06/27 03:32:54 [alert] 1199#0: worker process 1203 exited on signal 9
2012/06/27 03:32:54 [alert] 1199#0: worker process 1200 exited on signal 9
2012/06/27 03:32:54 [alert] 1199#0: worker process 1201 exited on signal 9
2012/06/27 03:32:54 [alert] 1199#0: worker process 1202 exited on signal 9


[email protected]:~# cat /var/log/nginx/access.log
31.210.99.87 - - [27/Jun/2012:09:09:08 +0400] "GET /w00tw00t.at.ISC.SANS.DFind:) HTTP/1.1" 400 172 "-" "-"
88.191.138.103 - - [27/Jun/2012:13:27:08 +0400] "GET /cms/cmx.jsp HTTP/1.1" 301 184 "-" "-"
88.191.138.103 - - [27/Jun/2012:13:27:08 +0400] "GET /iesvc/iesvc.jsp HTTP/1.1" 301 184 "-" "-"
88.191.138.103 - - [27/Jun/2012:13:27:08 +0400] "GET /cmd2/index.jsp HTTP/1.1" 301 184 "-" "-"
88.191.138.103 - - [27/Jun/2012:13:27:09 +0400] "GET /cmd/index.jsp HTTP/1.1" 301 184 "-" "-"
58.97.147.197 - - [27/Jun/2012:17:17:19 +0400] "GET / HTTP/1.1" 301 184 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X     10_7_4) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5"
58.97.147.197 - - [27/Jun/2012:17:17:37 +0400] "GET / HTTP/1.1" 301 184 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5"
58.97.147.197 - - [27/Jun/2012:17:17:38 +0400] "-" 400 0 "-" "-"
58.97.147.197 - - [27/Jun/2012:17:17:38 +0400] "-" 400 0 "-" "-"
58.97.147.197 - - [27/Jun/2012:17:17:48 +0400] "-" 400 0 "-" "-"

-

[email protected]:~# cat  /var/log/daemon.log
Jun 26 20:48:10 server xinetd[1177]: Exiting...
Jun 26 20:51:58 server xinetd[1174]: Reading included configuration file: /etc/xinetd.d/daytime [file=/etc/xinetd.d/daytime] [line=28]
Jun 26 20:51:58 server xinetd[1174]: Reading included configuration file: /etc/xinetd.d/discard [file=/etc/xinetd.d/discard] [line=26]
Jun 26 20:51:58 server xinetd[1174]: Reading included configuration file: /etc/xinetd.d/echo [file=/etc/xinetd.d/echo] [line=25]
Jun 26 20:51:58 server xinetd[1174]: Reading included configuration file: /etc/xinetd.d/time [file=/etc/xinetd.d/time] [line=26]
Jun 26 20:51:58 server xinetd[1174]: removing chargen
Jun 26 20:51:58 server xinetd[1174]: removing chargen
Jun 26 20:51:58 server xinetd[1174]: removing daytime
Jun 26 20:51:58 server xinetd[1174]: removing daytime
Jun 26 20:51:58 server xinetd[1174]: removing discard
Jun 26 20:51:58 server xinetd[1174]: removing discard
Jun 26 20:51:58 server xinetd[1174]: removing echo
Jun 26 20:51:58 server xinetd[1174]: removing echo
Jun 26 20:51:58 server xinetd[1174]: removing time
Jun 26 20:51:58 server xinetd[1174]: removing time
Jun 26 20:51:58 server xinetd[1174]: xinetd Version 2.3.14 started with libwrap loadavg options compiled in.
Jun 26 20:51:58 server xinetd[1174]: Started working: 0 available services
Jun 26 20:52:01 server vnstatd[1330]: vnStat daemon 1.11 started.
Jun 26 20:52:01 server vnstatd[1330]: Monitoring: venet0 
Jun 27 00:23:41 server xinetd[1174]: Exiting...
Jun 27 02:21:12 server vnstatd[1349]: vnStat daemon 1.11 started.
Jun 27 02:21:12 server vnstatd[1349]: Monitoring: venet0 
Jun 27 03:20:44 server xinetd[1166]: attribute: disable should not be in default section [file=/etc/xinetd.conf] [line=12]
Jun 27 03:20:44 server xinetd[1166]: Reading included configuration file: /etc/xinetd.d/chargen [file=/etc/xinetd.conf] [line=15]
Jun 27 03:20:44 server xinetd[1166]: Reading included configuration file: /etc/xinetd.d/daytime [file=/etc/xinetd.d/daytime] [line=28]
Jun 27 03:20:44 server xinetd[1166]: Reading included configuration file: /etc/xinetd.d/discard [file=/etc/xinetd.d/discard] [line=26]
Jun 27 03:20:44 server xinetd[1166]: Reading included configuration file: /etc/xinetd.d/echo [file=/etc/xinetd.d/echo] [line=25]
Jun 27 03:20:44 server xinetd[1166]: Reading included configuration file: /etc/xinetd.d/time [file=/etc/xinetd.d/time] [line=26]
Jun 27 03:20:44 server xinetd[1166]: removing chargen
Jun 27 03:20:44 server xinetd[1166]: removing chargen
Jun 27 03:20:44 server xinetd[1166]: removing daytime
Jun 27 03:20:44 server xinetd[1166]: removing daytime
Jun 27 03:20:44 server xinetd[1166]: removing discard
Jun 27 03:20:44 server xinetd[1166]: removing discard
Jun 27 03:20:44 server xinetd[1166]: removing echo
Jun 27 03:20:44 server xinetd[1166]: removing echo
Jun 27 03:20:44 server xinetd[1166]: removing time
Jun 27 03:20:44 server xinetd[1166]: removing time
Jun 27 03:20:44 server xinetd[1166]: xinetd Version 2.3.14 started with libwrap loadavg options compiled in.
Jun 27 03:20:44 server xinetd[1166]: Started working: 0 available services
Jun 27 03:20:46 server vnstatd[1249]: vnStat daemon 1.11 started.
Jun 27 03:20:46 server vnstatd[1249]: Monitoring: venet0 
Jun 27 03:32:41 server xinetd[1166]: Exiting...
Jun 27 03:33:32 server vnstatd[1380]: vnStat daemon 1.11 started.
Jun 27 03:33:32 server vnstatd[1380]: Monitoring: venet0 
[email protected]:~# 

-

Anything else you need let me know

© Server Fault or respective owner

Related posts about mysql

Related posts about vps