Can search engine robots read file with permission 640?

Posted by dkjain on Pro Webmasters See other posts from Pro Webmasters or by dkjain
Published on 2012-12-13T12:51:15Z Indexed on 2012/12/13 17:17 UTC
Read the original article Hit count: 361

Filed under:

robots.txt

|

serps

I am on a shared web hosting linux server. I want search engine robots/spiders to be able to read the robots.txt but not any one typing www.mysite.com/robots.txt.

As per the following google group post, the user specifies that by setting file permission to 640, it's possible to deny access to robots.txt file by the world but still enable search engine robots to read them.

Is that true? If not how it's possible to deny general public access to robots.txt but still allow Search engine robots to read them.

© Pro Webmasters or respective owner

Related posts about robots.txt

Robots.txt practices with .htaccess redirections (inherits)

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place. We have a hosting account that enacts primary and add-on domains. All of our domains and subdomains, including the primary domain, is redirected via htaccess 301s to their own subdirectories… >>> More
mod evasive not working properly on ubuntu 10.04

as seen on Server Fault - Search for 'Server Fault'
I have an ubuntu 10.04 server where I installed mod_evasive using apt-get install libapache2-mod-evasive I already tried several configurations, the result stays the same. The blocking does work, but randomly. I tried with low limis and long blocking periods as well as short limits. The behaviour… >>> More
Cross-domain jQuery using YQL gives robots.txt error

as seen on Stack Overflow - Search for 'Stack Overflow'
On the page http://qxlapps.dk/test.htm I am trying to perform an Ajax load from another domain, qxlapp.dk. I am using James Padolsey's xdomainajax.js plugin from: http://james.padolsey.com/javascript/cross-domain-requests-with-jquery/ When I open my test page, I get no output, but FireBug shows… >>> More
Asterisk in robots.txt

as seen on Stack Overflow - Search for 'Stack Overflow'
Wondering if following will work for google in robots.txt Disallow: /*.action I need to exclude all urls ending with .action. Is this correct? >>> More
SEO chaos from changing robots.txt file in Wordpress site

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi there, I recently edited the robots.txt file in my site using a wordpress plugin. However, since i did this, google seems to have removed my site from their search page. I'd appreciate if I could get an expert opinion on why this is so, and a possible solution. I'd initially done it to increase… >>> More

Related posts about serps

Why are intermittent pages disappearing from SERPs?

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have a very basical informational site with about 50+ pages. Specifically, there is a page about each site. I have been tracking pages daily for months and they have been streadily increasing in the SERPs. Suddenly, about 3-4 weeks ago, about 20 pages were completely missing from the SERPS, though… >>> More
Why Does the Same Site in Different SERPs Contain and Not Contain Google Sitelinks

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Is there a way to get sitelinks on a Google SERP when searching on a site's name vs. the sites' web address? Example, if you search "twin city kings", the first result is the website for Twin City Kings without sitelinks. But if you search "twincitykings.com", the first result is the website for… >>> More
Multiple site URLs now showing up in SERPs

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Recently when you search for me on Google several URLs from my blog have started showing up, instead of just the main URL. 1) https://scotthelme.co.uk/ 2) https://scotthelme.co.uk/category/wifi-pineapple-2/? 3) https://scotthelme.co.uk/wifi-pineapple-karma-sslstrip/ Is there a reason this starts… >>> More
Informing Googlebot for deprecated pages

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I publish timetables in my website. For example last year I published Number 2 bus Summer 2013 timetable. I has pretty good ranking on Google SERPs for number 2 bus timetable But this year I added a new page with the name "Number 2 bus Summer 2014 timetable". When users search number 2 bus timetable… >>> More
Did anyone experience negative SERP movement after implementing rel=author?

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I am not intrested in why I don't see the picture in SERPs. So I know this is borderline off-limits but I turned every stone in the web (including DDgo) trying to find anybody experiencing a worse position in SERPs after implementing rel=author tags. In Google Webmaster Tools: Everything seems… >>> More