robots.txt dissalow url containing string with a '/' at the end

Posted by thanili on Pro Webmasters See other posts from Pro Webmasters or by thanili
Published on 2013-11-11T08:40:40Z Indexed on 2013/11/11 10:27 UTC
Read the original article Hit count: 305

Filed under:

google-webmaster-tools

|

web-crawlers

|

robots.txt

i have a website with thousands of dynamic pages. I want to use the robots.txt file in order to dissalow certain url patterns corresponding to pages with duplicate content.

For example i have a page for article itemA belonging to category catA/subcatA, with URL:

/catA/subcatA/itemA

this is the URL that i want to be indexed from google.

this article is also visible via tagging in various other places in the web site. The URLs produced via tagging is like:

/tagA1/itemA

this URL i want NOT to be indexed from google. However i want to have indexed all tag listings:

/tagA1

so how can i achieve this? dissalow URLs of including a specific string with a '/' at the end?

/tagA1/ itemA - dissalow

/tagA1 - allow

© Pro Webmasters or respective owner

Related posts about google-webmaster-tools

Weird entry for robots.txt on a Naked Domain in Google Webmaster Tools

as seen on Pro Webmasters - Search for 'Pro Webmasters'
We own a .co.uk address and use an Internet hosting company that has made mistakes around DNS in the past. Our main site is hosted on www. and their reluctance to allow editing of AAAA records on-line means our naked domain does not resolve. Currently when we attempt to reach the naked version there… >>> More
How to submit sitemap when your website has partial https? - Error: "Not in Domain"

as seen on Pro Webmasters - Search for 'Pro Webmasters'
My website is an ecommerce that is set up to do http for the item browsing portion, but https for things like shopping cart, contact us, etc.. (anything that has forms on it). I've submitted my website a long time ago to google webmaster tools as http://www.mywebsite.com. I also submitted a sitemap… >>> More
Why Use Google Webmaster Tools

as seen on Ezine Articles - Search for 'Ezine Articles'
This article talks about why Google Webmaster Tools is an amazing program for optimizing and debugging your website. It provides many advantages for any website owner who uses it. >>> More
What Are Soft 404s in Google Webmaster Tools

as seen on Ezine Articles - Search for 'Ezine Articles'
There are two types of 404 response codes when a web page cannot be found. Google Webmaster Tools now displays both types. Find out why you do not want to have any soft 404 errors. >>> More
Revamped Google Webmaster Tools

as seen on Get blogged by JoKi - Search for 'Get blogged by JoKi'
With a positive surprise I realized today that Google's Webmaster Tools had some minor overhauling and provide some more details than before. Most obvious are the changes on the dashboard where the Top Search Queries now provide information about impressions and clicktroughs instead of the rankings… >>> More

Related posts about web-crawlers

Discover How Affiliates Marketers Can Optimize Their Blog For Search Engine Web Crawlers - #1

as seen on Ezine Articles - Search for 'Ezine Articles'
Search Engine Optimization is a FREE way of internet marketing that all internet marketers should take full advantage of it. It is all about building links to your websites or blogs. There are 2 ways of doing it. >>> More
How do web crawlers affect site statistics?

as seen on Stack Overflow - Search for 'Stack Overflow'
What are ways in which web crawlers (both from search engines and non-search engines) could affect site statistics (e.g., when doing AB-testing different page variations)? And what are ways to take care of these problems? For example: Do a lot of people writing web crawlers often delete their cookies… >>> More
Is there a list of known web crawlers?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm trying to get accurate download numbers for some files on a web server. I look at the user agents and some are clearly bots or web crawlers, but many for many I'm not sure, they may or may not be a web crawler and they are causing many downloads so it's important for me to know. Is there somewhere… >>> More
Detecting 'stealth' web-crawlers

as seen on Stack Overflow - Search for 'Stack Overflow'
What options are there to detect web-crawlers that do not want to be detected? (I know that listing detection techniques will allow the smart stealth-crawler programmer to make a better spider, but I do not think that we will ever be able to block smart stealth-crawlers anyway, only the ones that… >>> More
How to best develop web crawlers

as seen on Stack Overflow - Search for 'Stack Overflow'
Heyall, I am used to create some crawlers to compile information and as I come to a website I need the info I start a new crawler specific for that site, using shell scripts most of the time and sometime PHP. The way I do is with a simple for to iterate for the page list, a wget do download it and… >>> More