What's the best way to write a maintainable web scraping app?

Posted by Benj on Stack Overflow See other posts from Stack Overflow or by Benj
Published on 2009-11-09T11:17:10Z Indexed on 2010/05/10 13:24 UTC
Read the original article Hit count: 214

I wrote a perl script a while ago which logged into my online banking and emailed me my balance and a mini-statement every day. I found it very useful for keeping track of my finances. The only problem is that I wrote it just using perl and curl and it was quite complicated and hard to maintain. After a few instances of my bank changing their webpage I got fed up of debugging it to keep it up to date.

So what's the best way of writing such a program in such a way that it's easy to maintain? I'd like to write a nice well engineered version in either Perl or Java which will be easy to update when the bank inevitably fiddle with their web site.

© Stack Overflow or respective owner

Related posts about screen-scraping

Related posts about perl