We have a production server that seems to Seg Fault a few times every day. The fault is picked up by Apache and logged in the error log - but there seems to be no traffic around the time. If it's a request generating the fault then it looks like it happens before any other logging is made so I can't see how it's happening so it's very hard to debug.
Our setup is Linux 64 bit Centos 5.3
Apache is loaded with the following modules
apachectl -t -D DUMP_MODULES | more
Loaded Modules:
core_module (static)
mpm_prefork_module (static)
http_module (static)
so_module (static)
auth_basic_module (shared)
auth_digest_module (shared)
authn_file_module (shared)
authn_alias_module (shared)
authn_anon_module (shared)
authn_dbm_module (shared)
authn_default_module (shared)
authz_host_module (shared)
authz_user_module (shared)
authz_owner_module (shared)
authz_groupfile_module (shared)
authz_dbm_module (shared)
authz_default_module (shared)
ldap_module (shared)
authnz_ldap_module (shared)
include_module (shared)
log_config_module (shared)
logio_module (shared)
env_module (shared)
ext_filter_module (shared)
mime_magic_module (shared)
expires_module (shared)
deflate_module (shared)
headers_module (shared)
usertrack_module (shared)
setenvif_module (shared)
mime_module (shared)
dav_module (shared)
status_module (shared)
autoindex_module (shared)
info_module (shared)
dav_fs_module (shared)
vhost_alias_module (shared)
negotiation_module (shared)
dir_module (shared)
actions_module (shared)
speling_module (shared)
userdir_module (shared)
alias_module (shared)
rewrite_module (shared)
proxy_module (shared)
proxy_balancer_module (shared)
proxy_ftp_module (shared)
proxy_http_module (shared)
proxy_connect_module (shared)
cache_module (shared)
suexec_module (shared)
disk_cache_module (shared)
file_cache_module (shared)
mem_cache_module (shared)
cgi_module (shared)
version_module (shared)
security2_module (shared)
unique_id_module (shared)
fcgid_module (shared)
php5_module (shared)
proxy_ajp_module (shared)
ssl_module (shared)
Here's an exert from the Apache error log:
[Mon Mar 15 06:39:25 2010] [error] [client 213.246.222.74] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
[Mon Mar 15 07:41:31 2010] [error] [client 213.246.222.74] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
[Mon Mar 15 08:24:16 2010] [error] [client 67.19.250.146] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
[Mon Mar 15 08:43:46 2010] [error] [client 213.246.222.74] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
[Mon Mar 15 08:54:02 2010] [error] [client 74.208.123.71] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
[Mon Mar 15 09:09:51 2010] [notice] child pid 2138 exit signal Segmentation fault (11), possible coredump in /tmp
[Mon Mar 15 09:45:27 2010] [error] [client 213.246.222.74] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
[Mon Mar 15 09:49:05 2010] [error] [client 190.12.113.196] File does not exist: /var/www/vhosts/default/htdocs/phpMyAdmin
[Mon Mar 15 09:49:06 2010] [error] [client 190.12.113.196] File does not exist: /var/www/vhosts/default/htdocs/PMA
And the Access log around the same time (09:09:51):
213.246.222.74 - - [15/Mar/2010:08:43:46 +0000] "GET /" 400 561 "-" "-"
208.80.193.28 - - [15/Mar/2010:08:52:20 +0000] "GET / HTTP/1.0" 301 313 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; SU 2.009)"
74.208.123.71 - - [15/Mar/2010:08:54:02 +0000] "GET /w00tw00t.at.ISC.SANS.DFind:) HTTP/1.1" 400 298 "-" "-"
81.149.146.231 - - [15/Mar/2010:09:15:18 +0000] "GET /zabbix/ HTTP/1.1" 200 3565 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; en-us) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10"
81.158.71.196 - - [15/Mar/2010:09:16:06 +0000] "GET / HTTP/1.1" 301 313 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.0.18) Gecko/2010020219 Firefox/3.0.18"
213.246.222.74 - - [15/Mar/2010:09:45:27 +0000] "GET /" 400 561 "-" "-"
213.246.222.74 - - [15/Mar/2010:09:45:27 +0000] "GET /w00tw00t.at.ISC.SANS.DFind:) HTTP/1.1" 400 298 "-" "-"
190.12.113.196 - - [15/Mar/2010:09:49:05 +0000] "GET /phpMyAdmin/main.php HTTP/1.0" 404 295 "-" "-"
So As you can see, there's no access logged around the time of the fault!! How annoying :s
I enabled core dumps and here is the backtrace:
#0 0x00007f9c8c8a858b in memcpy () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f9c8cfb066d in apr_pstrcat (a=<value optimized out>) at strings/apr_strings.c:165
cp = 0x1fa6b "\205¦H\211¦t`¦\003"
argp = 0x7f9c9ad790e8 "Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Re"...
res = 0x0
saved_lengths = {129643, 2, 43, 140310399395576, 0, 140310394592712}
nargs = <value optimized out>
len = <value optimized out>
adummy = {{gp_offset = 16, fp_offset = 32668, overflow_arg_area = 0x7fff968a0ec0, reg_save_area = 0x7fff968a0de0}}
#2 0x00007f9c8cfb1bf9 in apr_table_merge (t=0x7f9c8f83b148, key=0x7f9c85a465fe "Vary", val=0x7f9c9ad99070 "Referer, Referer, Referer, Referer, Referer") at tables/apr_tables.c:688
next_elt = (apr_table_entry_t *) 0x7f9c8f83b270
end_elt = (apr_table_entry_t *) 0x7f9c8f83b270
checksum = <value optimized out>
hash = 22
#3 0x00007f9c85a42cfa in ?? () from /etc/httpd/modules/mod_rewrite.so
No symbol table info available.
#4 0x00007f9c85a44022 in ?? () from /etc/httpd/modules/mod_rewrite.so
No symbol table info available.
#5 0x00007f9c8e87bd1a in ap_run_fixups () from /usr/sbin/httpd
No symbol table info available.
#6 0x00007f9c8e88e8f8 in ap_process_request () from /usr/sbin/httpd
No symbol table info available.
#7 0x00007f9c8e88bb40 in ?? () from /usr/sbin/httpd
No symbol table info available.
#8 0x00007f9c8e887ca2 in ap_run_process_connection () from /usr/sbin/httpd
No symbol table info available.
#9 0x00007f9c8e892849 in ?? () from /usr/sbin/httpd
No symbol table info available.
#10 0x00007f9c8e892ada in ?? () from /usr/sbin/httpd
No symbol table info available.
#11 0x00007f9c8e892b90 in ?? () from /usr/sbin/httpd
No symbol table info available.
#12 0x00007f9c8e89387b in ap_mpm_run () from /usr/sbin/httpd
No symbol table info available.
#13 0x00007f9c8e86de48 in main () from /usr/sbin/httpd
No symbol table info available.
Can anyone shed any light on how to move forward with this? I can confirm that the server is operational and doesn't appear to be misbehaving - the failures are so infrequent that I haven't seen it do one while making a request myself.
Really appreciate any help! Cheers!
