How to protect/monitor your site from crawling by malicious user
- by deathy
Situation:
Site with content protected by username/password (not all controlled since they can be trial/test users)
a normal search engine can't get at it because of username/password restrictions
a malicious user can still login and pass the session cookie to a "wget -r" or something else.
The question would be what is the best solution to monitor such activity and respond to it (considering the site policy is no-crawling/scraping allowed)
I can think of some options:
Set up some traffic monitoring solution to limit the number of requests for a given user/IP.
Related to the first point: Automatically block some user-agents
(Evil :)) Set up a hidden link that when accessed logs out the user and disables his account. (Presumably this would not be accessed by a normal user since he wouldn't see it to click it, but a bot will crawl all links.)
For point 1. do you know of a good already-implemented solution? Any experiences with it? One problem would be that some false positives might show up for very active but human
users.
For point 3: do you think this is really evil? Or do you see any possible problems with it?
Also accepting other suggestions.
