Quick Question, robots.txt Disallow: /*/ does what exactly?
- by Exit
A SEO firm suggested changing the robots.txt to:
User-agent: *
Disallow: /*/
Allow: /ims/
I'm not sure what that would do, but my guess is that is would tell all robots to index nothing but the ims folder. I understand the wildcard, but I'm confused by the slashes and don't know how they would play out in conjunction with the wildcard.
* Update *
I didn't mention that there is a sitemap listed in the robots.txt file, but according to one tech blogger, he realized that sitemaps trump robots exclusions. So, even though this says in Google Webmaster Tools that everything with a trailing slash will not be indexed, the sitemap contains the important links.
I did notice that the link count on Google went from 360 to 336, and the sitemap links under the URL scaled back to 3 from 6. I'm not sure the cause or what links were removed, though. Perhaps it cleaned out garbage.
I'm still clueless why they would add in 'Allow: /ims/', that seems pointless.
And a quick list of what would index according to the robots rules above (withouth the sitemap) using /*/:
domain.com Indexed
domain.com/page.html Indexed
domain.com/folder/ Not Indexed
domain.com/folder/page.html Not Indexed
