Issue with apache mod proxy, it stops proxying requests after a graceful restart but not all the time.
It seems to happen only on a Sunday when a graceful restart is triggered by logrotate.
[Sun Sep 9 05:25:06 2012] [notice] SIGUSR1 received. Doing graceful restart
[Sun Sep 9 05:25:06 2012] [notice] Apache/2.
2.22 (Ubuntu) Phusion_Passenger/3.0.11 configured -- resuming normal operations
[Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(492) failed in child 26153 for worker proxy:reverse
[Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(486) failed in child 26153 for worker http://api.myservice.org/api
[Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(487) failed in child 26153 for worker http://api.myservice.org/editor/$1
[Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(489) failed in child 26153 for worker http://api.myservice.org/build
[Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(490) failed in child 26153 for worker http://api.myservice.org/help
[Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(491) failed in child 26153 for worker http://api.myservice.org/motd.html
[Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(480) failed in child 26153 for worker http://api.myservice.org/api
[Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(481) failed in child 26153 for worker http://api.myservice.org/editor/$1
[Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(483) failed in child 26153 for worker http://api.myservice.org/build
[Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(484) failed in child 26153 for worker http://api.myservice.org/help
[Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(485) failed in child 26153 for worker http://api.myservice.org/motd.html
[Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(479) failed in child 26153 for worker http://api.myservice.org/motd.html
After these lines, the logs are flooded with 404's because the requests are not being proxied.
It's worth noting that the destination is just another vhost on the same apache instance, but the vhost (http://api.myservice.org) is serving passenger (mod_rails)
I was thinking that maybe there's some startup issues with the passenger workers not being ready during a graceful restart?
After a full restart resolves it and everything returns to normal.
//Edit
Here's the vhost config, thanks :)
<VirtualHost *:80>
UseCanonicalName Off
LogFormat "%V %h %l %u %t \"%r\" %s %b" vcommon
<Directory /var/www/vhosts>
RewriteEngine on
AllowOverride All
</Directory>
RewriteEngine on
RewriteCond /var/www/vhosts/%{SERVER_NAME} !-d
RewriteCond /var/www/vhosts/%{SERVER_NAME} !-l
RewriteRule ^ http://sitenotfound.myservice.org/ [R=302,L]
VirtualDocumentRoot /var/www/vhosts/%0/current
# Rewrite requests to /assets to map to the /var/file-store/<SERVER_NAME>/
RewriteMap lowercase int:tolower
RewriteCond %{REQUEST_URI} ^/assets/
RewriteRule ^/assets/(.*)$ /var/file-store/${lowercase:%{SERVER_NAME}}/$1
# Map /login to /editor.html as it's far friendlier.
RewriteCond %{REQUEST_URI} ^/login
RewriteRule .* /editor.html [PT]
# Forward some requests to the
API
ProxyPass /api http://api.myservice.org/api
ProxyPass /site.json http://api.myservice.org/api/editor/site
ProxyPassMatch ^/editor/(.*)$ http://api.myservice.org/editor/$1
ProxyPassMatch ^/api/(.*) http://api.myservice.org/api/$1
ProxyPass /build http://api.myservice.org/build
ProxyPass /help http://api.myservice.org/help
ProxyPass /motd.html http://api.myservice.org/motd.html
<Proxy *>
Order allow,deny
Allow from all
</Proxy>
# TODO generate slightly more specific Error Documents for 401/403/500's,
# but for now the 404 page is good enough
ErrorDocument 401 /404.html
ErrorDocument 403 /404.html
ErrorDocument 404 /404.html
ErrorDocument 500 /404.html
</VirtualHost>
