Googlebot cant access my site webmaster tools reply Unreachable robots.txt

Posted by Ahmad Ahmadi on Server Fault See other posts from Server Fault or by Ahmad Ahmadi
Published on 2013-10-27T06:15:43Z Indexed on 2013/10/27 9:57 UTC
Read the original article Hit count: 395

Filed under:
|
|

When I try to fetch my site as a googlebot in webmaster tools it return Unreachable robots.txt, after investigate I understood google bot can see my server: tcpdump | grep google it return that google can access my server with IP 66.249.81.172 or 66.249.75.111. but there is not any think in access log or error log or other apache logs. cat access_log | grep google or cat error_log | grep 66.249.81.172 Other bot (bing,...) can access apache but google cant. there is not any problem in my robots.txt or its permissions because as you know robots.txt is not necessary so I delete it but again webmaster tools returned Unreachable robots.txt not 404 not found! information about server: Server OS : CentOS 6 Web Server : Apache 2.x Firewall : IPTables is stoped SELinux is Disabled There is not any think else for security on my server. how can I investigate the problem and is there any other command that can help me to find the problem.

© Server Fault or respective owner

Related posts about apache2

Related posts about robots.txt