How would you protect a database of links from being scraped?

Posted by Yegor on Stack Overflow See other posts from Stack Overflow or by Yegor
Published on 2010-03-22T21:42:57Z Indexed on 2010/03/22 21:51 UTC
Read the original article Hit count: 344

I have a large database of links, which are all sorted in specific ways and are attached to other information, which is valuable (to some people).

Currently my setup (which seems to work) simply calls a php file like link.php?id=123, it logs the request with a timestamp into the DB. Before it spits out the link, it checks how many requests were made from that IP in the last 5 minutes. If its greater than x, it redirects you to a captcha page.

That all works fine and dandy, but the site has been getting really popular (as well as been getting DDOsed for about 6 weeks), so php has been getting floored, so Im trying to minimize the times I have to hit up php to do something. I wanted to show links in plain text instead of thru link.php?id= and have an onclick function to simply add 1 to the view count. Im still hitting up php, but at least if it lags, it does so in the background, and the user can see the link they requested right away.

Problem is, that makes the site REALLY scrapable. Is there anything I can do to prevent this, but still not rely on php to do the check before spitting out the link?

© Stack Overflow or respective owner

Related posts about php

Related posts about screen-scraping