Which token from a long User-Agent should I use in robots.txt?
- by Gaia
The definition of User-Agent states that several tokens can be included, as deemed necessary by the client.
I want to block certain bots via robots.txt and I am confused as to which part of the User-Agent string to use, especially for more obscure bots. For example:
Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: [email protected])"
JS-Kit URL Resolver, http://js-kit.com/
Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html
Do I use the second token? Can tokens contain spaces, or did the SEOkicks folks forget a semicolon after SEOkicks-Robot? I don't actually intend on making my question specific to a couple bots - I want to know the guideline: which part of UA do I place in robots.txt for these exotic bots with UA as long as a haiku?
User-agent: uMBot-LN/1.0
Disallow: /
PS: Thank you but I do not need to hear that undesirable bots are better blocked with mod_security. I already have commercial mod_sec rules in place.
