Google Sitemap and Robots.txt Issue

Posted by Sarfaraz Soomro on Stack Overflow See other posts from Stack Overflow or by Sarfaraz Soomro
Published on 2009-09-28T08:16:17Z Indexed on 2010/03/27 21:03 UTC
Read the original article Hit count: 316

Filed under:

google

|

sitemap

|

robots.txt

Hi,

We have a sitemap at our site, http://www.gamezebo.com/sitemap.xml

Some of the urls in the sitemap, are being reported in the webmaster central as being blocked by our robots.txt, see, gamezebo.com/robots.txt ! Although these urls are not Disallowed in Robots.txt. There are other such urls aswell, for example, gamezebo.com/gamelinks is present in our sitemap, but it's being reported as "URL restricted by robots.txt".

Also I have this parse result in the Webmaster Central that says, "Line 21: Crawl-delay: 10 Rule ignored by Googlebot". What does it mean?

I appreciate your help,

Thanks.

© Stack Overflow or respective owner

Related posts about google

Removing malware of a particular kind

as seen on Super User - Search for 'Super User'
I need to remove some malware from my computer. It is a trojan, and very annoying. It blocks access to Google and search sites. The trojan, with its name spelled out on each line cause it seems to block sites when i reference it in a url, is a r t (some text to mess it up) e m (more text i s First… >>> More
Trouble installing Matlab

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
So whenever I try to install Matlab, the first image appears, but then goes away and that's the end of it. Here's what I'm doing- user@host$> cd ~/mathworks_downloads user@host$> unzip matlab_R2012a_student_glnx86_installer.zip user@host$> ./install The Matlab logo and image pop up here… >>> More
Google chrome is always searching in local google domain instead of Google.com

as seen on Super User - Search for 'Super User'
I have changed in the searched preferences to google.com but still when I do search from the address bar (instant or non-instant) it will go to google.co.kr. Even though I change "Google.com in English", still same... The only way is to open google.com website first, then do search in it. So the… >>> More
Google I/O 2010: Google TV Keynote - Introducing Google TV

as seen on Google Code - Search for 'Google Code'
Google I/O 2010: Google TV Keynote - Introducing Google TV Due to licensing and permissions issues, we are unable to show the full Google TV demonstration from the Day 2 keynote at Google I/O. Until we are able to get these permissions, please check out these clips. For Google I/O session videos… >>> More
Google I/O 2010: Google TV Keynote - Android Apps On Google TV

as seen on Google Code - Search for 'Google Code'
Google I/O 2010: Google TV Keynote - Android Apps On Google TV Due to licensing and permissions issues, we are unable to show the full Google TV demonstration from the Day 2 keynote at Google I/O. Until we are able to get these permissions, please check out these clips. For Google I/O session videos… >>> More

Related posts about sitemap

problem with Webmaster Google Sitemap

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have a wp mu 3.6.1 with domain mapping (0.5.4.3) with w3tc (0.9.3) and Google XML Sitemaps (4.0 BETA). I have 4 different sitemaps. sub-1.com/sitemap.xml sub-2.com/sitemap.xml sub-3.com/sitemap.xml sub-4.com/sitemap.xml on google webmaster i got 59 errors & 14 warnings. Sitemap errorsErrors: We… >>> More
The Sitemap Paradox

as seen on Pro Webmasters - Search for 'Pro Webmasters'
We use a sitemap on Stack Overflow, but I have mixed feelings about it. Web crawlers usually discover pages from links within the site and from other sites. Sitemaps supplement this data to allow crawlers that support Sitemaps to pick up all URLs in the Sitemap and learn about those URLs using… >>> More
Sitemaps - do I need to submit each sitemap in sitemap_index.xml to Google Webmaster tools?

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I am having a Wordpress blog on my CentOS server. There is no sitemap.xml in the root directory but there is sitemap_index.xml file in the root directory which contains the following code: <?xml-stylesheet type="text/xsl" href="http://mywebsite.com/wp-content/plugins/wordpress-seo/css/xml-sitemap-xsl… >>> More
Yahoo sitemap validation

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Hello, I am trying to submit sitemap.xml (index) to Yahoo Site Explorer but with no luck. I tried using website feed option in the site explorer to submit the sitemap, but I got validation errors. However, when submitting the same sitemap to google webmaster tools, the sitemap was validated successfully… >>> More
Modify code to change timestamp timezone in sitemap

as seen on Stack Overflow - Search for 'Stack Overflow'
Below is the code from a plugin I use for sitemaps. I would like to know if there's a way to "enforce" a different timezone on all date variable and functions in it. If not, how do I modify the code to change the timestamp to reflect a different timezone? Say for example, to America/New_York timezone… >>> More