I am in charge of production servers serving static content for a website. Those servers are constantly being crawled by bots looking for potential exploits (which isn't that much of a problem security-wise because no application can be reached behind the web server) but
generates thousands of 404 per day, sometimes per hour. I am looking into ways of blocking those requests but it's tricky (you want to make sure you don't block legitimate traffic and these bots are becoming more and more clever at looking like they're legit) and is going to take me a while to find an acceptable solution.
In the meantime I would like to reduce the performance impact of serving those 404 pages. Indeed we're using nginx which by default is configured to serve it's 404 page from the disk (This can be changed using the error_page directive but in the end the 404 will either have to be served from disk or from another external source (e.g. upstream application which would be worst)) which isn't ideal.
I ran a test with ab on my local machine with a basic configuration: in one case I echo a message directly from nginx so the disk isn't touched at all, in the other case I hit a missing page and nginx serves its 404 from disk.
server {
# [...] the default nginx stuff
location / {
}
location /this_page_exists {
echo "this page was found";
}
}
Here are the test results (my laptop has Intel(R) Core(TM) i7-2670QM + SSD in case you're wondering why they are so high):
$ ab -n 500000 -c 1000 http://localhost/this_page_exists
Requests per second: 25609.16 [#/sec] (mean)
$ ab -n 500000 -c 1000 http://localhost/this_page_doesnt_exists
Requests per second: 22905.72 [#/sec] (mean)
As you can see, returning a value with echo is 11% ((25609-22905)÷22905×100) faster than serving the 404 page from disk. Accordingly I would like to echo a simple 404 Page not Found string from nginx.
I tried many things so far but they all failed, essentially the idea was this:
location / {
try_files $uri @not_found;
}
location @not_found {
echo "404 - Page not found";
}
The problem is that as soon as the echo directive is used, the http response code is set to 200. I tried changing that by doing error_page 200 = 400 but that breaks the configuration.
How can I serve a 404 page directly from nginx? (without hacking the source which may be might next step)
