I got a site server.com which I want to be returned when it is directly adressed using server.com or www.server.com but issue a 404 if the webserver gets accessed using its ip or another hostname.

I got something like the following:

NameVirtualhost 10.0.0.1:80
<VirtualHost 10.0.0.1:80> #This should be used if Host is server.com or www.server.com
ServerName server.com
ServerAlias server.com www.server.com
DocumentRoot /var/www/
</VirtualHost>

<VirtualHost 10.0.0.1:80> #This should be used alternatively
RewriteEngine On
RewriteRule (.*) - [R=404]
</VirtualHost>

How can I now set that the second entry should be used if hostname is not server.com or www.server.com?