Is there a way to disallow crawling of only HTTPS in robots.txt?

Posted by David Wilkins on Pro Webmasters See other posts from Pro Webmasters or by David Wilkins
Published on 2014-05-29T14:12:59Z Indexed on 2014/05/30 3:53 UTC
Read the original article Hit count: 231

Filed under:

robots.txt

|

ssl

I just realized that Bingbot is crawling my company's website's pages over https. Bing already crawls the site over http, so this seems frivolous. Is there a way to specify Disallow: / for https only?

According to Wikipedia, each protocol has its own robots.txt

And according to Google's Robots.txt Specification, the robots.txt applies to http AND https

I don't want to Disallow: / for Bing totally, just over https.

© Pro Webmasters or respective owner

Related posts about robots.txt

Robots.txt practices with .htaccess redirections (inherits)

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place. We have a hosting account that enacts primary and add-on domains. All of our domains and subdomains, including the primary domain, is redirected via htaccess 301s to their own subdirectories… >>> More
mod evasive not working properly on ubuntu 10.04

as seen on Server Fault - Search for 'Server Fault'
I have an ubuntu 10.04 server where I installed mod_evasive using apt-get install libapache2-mod-evasive I already tried several configurations, the result stays the same. The blocking does work, but randomly. I tried with low limis and long blocking periods as well as short limits. The behaviour… >>> More
Cross-domain jQuery using YQL gives robots.txt error

as seen on Stack Overflow - Search for 'Stack Overflow'
On the page http://qxlapps.dk/test.htm I am trying to perform an Ajax load from another domain, qxlapp.dk. I am using James Padolsey's xdomainajax.js plugin from: http://james.padolsey.com/javascript/cross-domain-requests-with-jquery/ When I open my test page, I get no output, but FireBug shows… >>> More
Asterisk in robots.txt

as seen on Stack Overflow - Search for 'Stack Overflow'
Wondering if following will work for google in robots.txt Disallow: /*.action I need to exclude all urls ending with .action. Is this correct? >>> More
SEO chaos from changing robots.txt file in Wordpress site

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi there, I recently edited the robots.txt file in my site using a wordpress plugin. However, since i did this, google seems to have removed my site from their search page. I'd appreciate if I could get an expert opinion on why this is so, and a possible solution. I'd initially done it to increase… >>> More

Related posts about ssl

Plesk SSL Certificate (Default cert when SSL enabled, CORRECT cert when SSL is disabled)

as seen on Server Fault - Search for 'Server Fault'
I'm running Plesk 8.6.0: I have an SSL cert installed through Plesk's admin interface. But I have a bit of an issue: When I enabled SSL for the site, and selected my cert, then restart httpd, Plesk defaults to using my self-signed default certificate. Conversely, when I disable SSL support for the… >>> More
apache renew ssl not working [on hold]

as seen on Server Fault - Search for 'Server Fault'
Downloaded a new ssl cert from go daddy and installed the cert on apache2 server put the cert in /etc/ssl/certs/ folder put the gd_bundle.crt in the /etc/ssl/ folder private key is in /etc/ssl/private/private.key I just replaced the original files with the new files, did not replace the private… >>> More
Cant connect to mysql using self signed SSL certificate

as seen on Server Fault - Search for 'Server Fault'
After creating a self-signed SSL certificate, I have configured my remote mysqld to use them (and ssl is enabled) I ssh into my remote server, and try connecting to its own mysqld using ssl (mysql server is 5.5.25).. ~> mysql -u <user> -p --ssl=1 --ssl-cert=client.cert --ssl-key=client… >>> More
How to log invalid client SSL certificate in SSL

as seen on Server Fault - Search for 'Server Fault'
I have a IIS web site which requires client certificate. I have turned off CRL checking. The client is unable to access the web site - he gets 403.17 (certificate expired) error. I would like to log the certificate he is using, becaue I think he is using the wrong certificate. Is there a way to… >>> More
Mixing SSL and non-SSL content in an Apache2 virtual host

as seen on Server Fault - Search for 'Server Fault'
I have a (hopefully) common scenario for one of my sites that I just can't seem to figure out how to deploy correctly. I have the following site and directories for example.com: These need to require SSL: /var/www/example.com/admin /var/www/example.com/order These need to be non-SSL: /var/www/example… >>> More