Googlebot cant access my site webmaster tools reply Unreachable robots.txt
        Posted  
        
            by 
                Ahmad Ahmadi
            
        on Server Fault
        
        See other posts from Server Fault
        
            or by Ahmad Ahmadi
        
        
        
        Published on 2013-10-27T06:15:43Z
        Indexed on 
            2013/10/27
            9:57 UTC
        
        
        Read the original article
        Hit count: 507
        
When I try to fetch my site as a googlebot in webmaster tools it return Unreachable robots.txt, after investigate I understood google bot can see my server: tcpdump | grep google it return that google can access my server with IP 66.249.81.172 or 66.249.75.111. but there is not any think in access log or error log or other apache logs. cat access_log | grep google or cat error_log | grep 66.249.81.172 Other bot (bing,...) can access apache but google cant. there is not any problem in my robots.txt or its permissions because as you know robots.txt is not necessary so I delete it but again webmaster tools returned Unreachable robots.txt not 404 not found! information about server: Server OS : CentOS 6 Web Server : Apache 2.x Firewall : IPTables is stoped SELinux is Disabled There is not any think else for security on my server. how can I investigate the problem and is there any other command that can help me to find the problem.
© Server Fault or respective owner