How can i verify a complete website's link(mostly links that are redirecting to 404 page) by using selenium Rc. Previously i tried to do this thong by using xenu and web link validator but in there results most of the links are showing 500 internal serevr error.and for the pages they are showing 500 internal server error the pages actuallt don't exists in the web site. So what is the concept if we can crawl through the website using selenium rc.?