Robots.txt syntax
- by Sinan
I not expert on robots.txt and i have the following in one of my clients robots.txt
User-agent: *
Disallow:
Disallow: /backup/
Disallow: /stylesheets/
Disallow: /admin/
I am not sure about the second line. Is this line disallows all spiders?