What is a good robots.txt for WP ?

Posted by Steven on Pro Webmasters See other posts from Pro Webmasters or by Steven
Published on 2010-12-13T11:10:29Z Indexed on 2011/01/12 16:00 UTC
Read the original article Hit count: 250

Filed under:

Wordpress

|

robots.txt

What is the "best" setup for robots.txt?
I'm using the following permalink structure in Wordpress: /%category%/%postname%/.

My robots.txt currently looks like this (copied from somewhere a long time ago):

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */comments

I want my comments to be indext. So I can remove this, right?
Do I want to disallow indexing categories because of my permalinkstructure?
An article can have several tags and be in multiple categories. This may cause duplicates in google. How should I work around this?

Would you change anything else here?

© Pro Webmasters or respective owner

Related posts about Wordpress

Add Your Own Domain to Your WordPress.com Blog

as seen on How to geek - Search for 'How to geek'
Now that you’ve got a nice blog on WordPress.com, why not get your own domain to brand your site? Here’s how you can easily register a new domain or move your existing domain to your WordPress site. By default, your free WordPress address is yourblog’sname.wordpress.com. But whether this… >>> More
Moving from wordpress.com to self-hosted wordpress blog

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, I have been writing articles on the wordpress.com blog, now i am looking to move it to self-hosted wordpress blog but i wonder: 1) Should i move all my articles on the new blog or just put an article on my last blog that more articles will be posted on my new blog?* 2) If i move all articles… >>> More
Include WordPress Content Outside of WordPress

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm looking to build a mobile version of a WordPress site and have most of it built in static files, but am trying to get access to the WordPress content from inside an external PHP file that the mobile site will run from. How can I get access to the loop or database without manually writing SQL queries… >>> More
What is the difference between theme making process using Wordpress Theme and Wordpress Theme Framew

as seen on Stack Overflow - Search for 'Stack Overflow'
What is the difference between "Wordpress Theme" and Wordpress Theme Frameworks? In terms of technicalities, Custom theme making workflow, File and folder structure, Anatomy. Wordpress default theme Vs Wordpress theme's framework's Child theme >>> More
Integration of WordPress With WordPress bbPress

as seen on Ezine Articles - Search for 'Ezine Articles'
bbPress is a powerful and fully featured forum software that can be integrated with blogging tools like WordPress. A first glance at the software, gives the impression that it is simple and devoid of any complicated structure, but one has to use it in order, to believe that it is a very powerful tool. >>> More

Related posts about robots.txt

Robots.txt practices with .htaccess redirections (inherits)

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place. We have a hosting account that enacts primary and add-on domains. All of our domains and subdomains, including the primary domain, is redirected via htaccess 301s to their own subdirectories… >>> More
mod evasive not working properly on ubuntu 10.04

as seen on Server Fault - Search for 'Server Fault'
I have an ubuntu 10.04 server where I installed mod_evasive using apt-get install libapache2-mod-evasive I already tried several configurations, the result stays the same. The blocking does work, but randomly. I tried with low limis and long blocking periods as well as short limits. The behaviour… >>> More
Cross-domain jQuery using YQL gives robots.txt error

as seen on Stack Overflow - Search for 'Stack Overflow'
On the page http://qxlapps.dk/test.htm I am trying to perform an Ajax load from another domain, qxlapp.dk. I am using James Padolsey's xdomainajax.js plugin from: http://james.padolsey.com/javascript/cross-domain-requests-with-jquery/ When I open my test page, I get no output, but FireBug shows… >>> More
Asterisk in robots.txt

as seen on Stack Overflow - Search for 'Stack Overflow'
Wondering if following will work for google in robots.txt Disallow: /*.action I need to exclude all urls ending with .action. Is this correct? >>> More
SEO chaos from changing robots.txt file in Wordpress site

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi there, I recently edited the robots.txt file in my site using a wordpress plugin. However, since i did this, google seems to have removed my site from their search page. I'd appreciate if I could get an expert opinion on why this is so, and a possible solution. I'd initially done it to increase… >>> More