Robot.txt can get all soft404s fixed?

Posted by olo on Pro Webmasters See other posts from Pro Webmasters or by olo
Published on 2014-06-10T01:00:47Z Indexed on 2014/06/10 3:43 UTC
Read the original article Hit count: 441

Filed under:

404

|

soft-404

I got many soft404 in Google webmaster Tools, and those webpages aren't existing any more. thus I am unable to insert <meta name="robots" content="noindex, nofollow"> into my pages, and I've been searching a while but didn't get some valuable clues.

There are about 100 URLs are soft 404, to redirect them all one by one is a bit silly as it would cost too much time for me.

If i just add those links into robot.txt like below

User-agent: *
Disallow: /mysite.asp
Disallow: /mysite-more.html

if this way will fix all soft404s solidly? or if there is a way to change all soft404 to hard404? Please give me some suggestions. Many thanks

© Pro Webmasters or respective owner

Related posts about 404

ubuntu/apt-get update said "Failed to Fetch http:// .... 404 not found"

as seen on Server Fault - Search for 'Server Fault'
Hi all, I'm trying to run apt-get update on ubuntu 9.10 I've configured my proxy server and I can access the internet without any problem: /etc/apt# wget "http://www.google.com" Resolving (...) Proxy request sent, awaiting response... 200 OK Length: 292 [text/html] Saving to: `index.html' 100%[=================================================================================================================================>]… >>> More
Cannot update, apt-get cannot fetch index files

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I have a fresh install of Ubuntu 11.10 from the iso 'ubuntu-11.10-desktop-amd64.iso'. I installed this in VMWare Fusion 4.1.1 running on OSX 10.7.3. When setting up the VM, I allowed easy install to take care of creating my user and installing VMWare tools. No problems during installation, everything… >>> More
Protecting Apache with Fail2Ban

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Having checked my Apache logs for the last two days I have noticed several attempts to access URLs such as /phpmyadmin, /phpldapadmin: 121.14.241.135 - - [09/Jun/2012:04:37:35 +0100] "GET /w00tw00t.at.blackhats.romanian.anti-sec:) HTTP/1.1" 404 415 "-" "ZmEu" 121.14.241.135 - - [09/Jun/2012:04:37:35… >>> More
Django flatpages raising 404 when DEBUG is False (404 and 500 templates exist)

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm using Django 1.1.1 stable. When DEBUG is set to True Django flatpages works correctly; when DEBUG is False every flatpage I try to access raises a custom 404 error (my error template is obviously working correctly). Searching around on the internet suggests creating 404 and 500 templates which… >>> More
x.gif in Apache logs

as seen on Server Fault - Search for 'Server Fault'
I manage a Django site where we host the media on a subdomain. There shouldn't be any requests for media to the main domain. However I keep seeing these requests for "x.gif" showing up in the access logs on the domain that's handled by WSGI (not the media domain). Can anyone explain what this is… >>> More

Related posts about soft-404

The number of soft 404 errors is increasing because of redirects to the home page

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have an increase in soft 404 errors. Using Apache in my .htaccess file I have: Redirect 301 /test.html? /page/pop/test Redirect 301 /about.html? /about I have also tried: Redirect 301 http://www.example.co.za/test.html? http://www.example.co.za/services/test however whenever I go to: http://www… >>> More
In Google webmaster tools, can a "soft 404" be triggered by the text on the page?

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I just ran across an error in Google Webmaster Tools that I have never seen before. I manage the website for my local community band (I play trombone). One of the pages on the site is a list of our upcoming performances. It is powered by a WordPress events plugin that uses a database of upcoming… >>> More
Why does Google report a soft 404 when I redirect to the signup page?

as seen on Pro Webmasters - Search for 'Pro Webmasters'
In the last month, I've got an increased number of "soft 404" errors reported by Google webmaster tools which actually work well for users. Configuration (maybe useless): I have a website built with rails 3.1 Authentication is handled by the gem Devise Problem: On this page http://en.bemyboat… >>> More
Google webmastertools soft 404 - How to update google search after updating it to 200

as seen on Pro Webmasters - Search for 'Pro Webmasters'
My site has many modules which are indexed by google. Recently there has been a database problem so the site was not appearing well like many links returned 404 i think. Now i have make it working and all the content what previously google indexed are as it were. How do we update google that i… >>> More
Soft 404 error on redirected outbound links

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have a redirect script on my site which sends visitors to an affiliate site. However in the last month I've noticed that Google webmaster tools is reporting my outbound links as a 404 error. Here is the breakdown on how its setup: My outbound links are coded like this: <a href="/f/c123" rel="nofollow"… >>> More