Googlebot cant access my site webmaster tools reply Unreachable robots.txt
- by Ahmad Ahmadi
When I try to fetch my site as a googlebot in webmaster tools it return Unreachable robots.txt, after investigate I understood google bot can see my server:
tcpdump | grep google
it return that google can access my server with IP 66.249.81.172 or 66.249.75.111.
but there is not any think in access log or error log or other apache logs.
cat access_log | grep google or cat error_log | grep 66.249.81.172
Other bot (bing,...) can access apache but google cant.
there is not any problem in my robots.txt or its permissions because as you know robots.txt is not necessary so I delete it but again webmaster tools returned Unreachable robots.txt not 404 not found!
information about server:
Server OS : CentOS 6
Web Server : Apache 2.x
Firewall : IPTables is stoped
SELinux is Disabled
There is not any think else for security on my server.
how can I investigate the problem and is there any other command that can help me to find the problem.
