Recovering from an incorrectly deployed robots.txt?
- by Doug T.
We accidentally deployed a robots.txt from our development site that disallowed all crawling. This has caused traffic to dip dramatically, and google results to report:
A description for this result is not available because of this site's robots.txt – learn more.
We've since corrected the robots.txt about a 1.5 weeks ago, and you can see our robots.txt here.
However, search results still report the same robots.txt message. The same appears to be true for Bing.
We've taken the following action:
Submitted site to be recrawled through google webmaster tools
Submitted a site map to google
(basically doing everything possible to say "Hey we're here! and we're crawlable!")
Indeed a lot of crawl activity seems to be happening lately, but still no description is crawled.
I noticed this question where the problem was specific to a 303 redirect back to a disallowed path.
We are 301 redirecting to /blog, but crawling is allowed here. This redirect is due to a site redesign, wordpress paths for posts such as /2012/02/12/yadda yadda have been moved to /blog/2012/02/12. We 301 redirect to wordpress for /blog to keep our google juice. However, the sitemap we submitted might have /blog URLs. I'm not sure how much this matters. We clearly want to preserve google juice for URLs linked to us from before our redesign with the /2012/02/... URLs.
So perhaps this has prevented some content from getting recrawled? How can we get all of our content, with links pointed to our site from pre-and-post redesign reporting descriptions? How can we resolve this problem and get our search traffic back to where it used to be?
