Nginx + PHP-FPM = "Random" 502 Bad Gateway
- by david
I am running Nginx and proxying php requests via FastCGI to PHP-FPM for processing. I will randomly receive 502 Bad Gateway error pages - I can reproduce this issue by clicking around my PHP websites very rapidly/refreshing a page for a minute or two. When I get the 502 error page all I have to do is refresh the browser and the page refreshes properly.
Here is my setup:
nginx/0.7.64 PHP 5.3.2 (fpm-fcgi)
(built: Apr 1 2010 06:42:04) Ubuntu
9.10 (Latest 2.6 Paravirt)
I compiled PHP-FPM using this ./configure directive
./configure --enable-fpm --sysconfdir=/etc/php5/conf.d --with-config-file-path=/etc/php5/conf.d/php.ini --with-zlib --with-openssl --enable-zip --enable-exif --enable-ftp --enable-mbstring --enable-mbregex --enable-soap --enable-sockets --disable-cgi --with-curl --with-curlwrappers --with-gd --with-mcrypt --enable-memcache --with-mhash --with-jpeg-dir=/usr/local/lib --with-mysql=/usr/bin/mysql --with-mysqli=/usr/bin/mysql_config --enable-pdo --with-pdo-mysql=/usr/bin/mysql --with-pdo-sqlite --with-pspell --with-snmp --with-sqlite --with-tidy --with-xmlrpc --with-xsl
My php-fpm.conf looks like this (the relevant parts):
...
<value name="pm">
<value name="max_children">3</value>
...
<value name="request_terminate_timeout">60s</value>
<value name="request_slowlog_timeout">30s</value>
<value name="slowlog">/var/log/php-fpm.log.slow</value>
<value name="rlimit_files">1024</value>
<value name="rlimit_core">0</value>
<value name="chroot"></value>
<value name="chdir"></value>
<value name="catch_workers_output">yes</value>
<value name="max_requests">500</value>
...
I've tried increasing the max_children to 10 and it makes no difference. I've also tried setting it to 'dynamic' and setting max_children to 50, and start_server to '5' without any difference.
I have tried using both 1 and 5 nginx worker processes.
My fastcgi_params conf looks like:
fastcgi_connect_timeout 60;
fastcgi_send_timeout 180;
fastcgi_read_timeout 180;
fastcgi_buffer_size 128k;
fastcgi_buffers 4 256k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
fastcgi_intercept_errors on;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_param REQUEST_URI $request_uri;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param DOCUMENT_ROOT $document_root;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx/$nginx_version;
fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;
fastcgi_param REDIRECT_STATUS 200;
Nginx logs the error as:
[error] 3947#0: *10530 connect()
failed (111: Connection refused) while
connecting to upstream, client:
68.40.xxx.xxx, server: www.domain.com, request: "GET /favicon.ico HTTP/1.1",
upstream: "fastcgi://127.0.0.1:9000",
host: "www.domain.com"
PHP-FPM logs the follow at the time of the error:
[NOTICE] pid 17161, fpm_unix_init_main(), line 255: getrlimit(nofile): max:1024, cur:1024
[NOTICE] pid 17161, fpm_event_init_main(), line 93: libevent: using epoll
[NOTICE] pid 17161, fpm_init(), line 50: fpm is running, pid 17161
[DEBUG] pid 17161, fpm_children_make(), line 403: [pool default] child 17162 started
[DEBUG] pid 17161, fpm_children_make(), line 403: [pool default] child 17163 started
[DEBUG] pid 17161, fpm_children_make(), line 403: [pool default] child 17164 started
[NOTICE] pid 17161, fpm_event_loop(), line 111: ready to handle connections
My CPU usage maxes out around 10-15% when I recreate the issue.
My Free mem (free -m) is 130MB
I had this intermittent 502 Bad Gateway issue when in was using php5-cgi to service my php requests as well. Does anyone know how to fix this?
