Do or can robots cause considerable performance issues?
- by Anicho
So the question in the title is exactly what I am trying to find out.
My case is:
At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website.
Out setup:
Lets say I have site www.mysite.co.uk this is a shop window to our online services which sit on www.mysiteonline.co.uk.
When people search in google for mysite they see mysiteonline.co.uk as well as mysite.co.uk.
Cases against stopping bots crawling:
We don't store gb's of data publicly available on the web
Most friendly bots, if they were to cause issues would have done so already
In our instance the bots can't crawl the site because it requires username & password
Stopping bots with robot .txt causes an issue with seo (ref.1)
If it was a malicious bot, it would ignore robot.txt or meta tags anyway
Ref 1.
If we were to block mysiteonline.co.uk from having robots crawl this will affect seo rankings and make it inconvenient for users who actively search for mysite to find mysiteonline. Which we can prove is the case for a good portion of our users.
