Robots Crawling Across Namespace?
- by Codex73
I migrated site from one domain to another.
Also placed permanent redirection on old account.
My stats logs are capturing this:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
/libro_metaboforte_chap5.php/members/members/file_chap6.php
I placed this on robots which wasn't present at time of migration.
Robots.txt Contents
User-agent: *
Allow: /
Disallow: /members/
Disallow: /includes/
HTACCESS FILE CONTENTS
DirectoryIndex index.php index.html
Options +FollowSymlinks
RewriteEngine On # Turn on the rewriting engine
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_URI} !^/store/?$
RewriteCond %{QUERY_STRING} !.
RewriteRule ^.+/?$ index.php [QSA,L]
RewriteCond %{QUERY_STRING} ^curlang=([a-z]*)$
RewriteRule ^.+/?$ index.php? [QSA,L]
Will continue to log incoming bot captures.
My htaccess does rewrite. I just added the robot file. The funny part is that is stepping in double directories...
I don't know if the problem was not having the 'robots.txt' in place or the actual in place htaccess doing rewrites?
