Optimising news fetching
- by aceBox
I have a web scraper for scraping news from different sources in wp7. My current appraoch for doing this is:
load newspapers information from xml file.
go to the specified sections and fetch the urls of the news items.
go to each url and fetch headline, image, publisher.
display using a MVVM architecture of windows phone.
The whole thing takes place asynchronously...meaning as soon as url from a section of a newspaper is fetched it is added to the queue, and the second stage consisting of fetching headline, image etc starts... and as soon this is fetched even for one article, it is displayed. Later on as more articles are fetched, they are added on to the list.
For the fetching purpose I am using a SmartThreadPool(http://www.codeproject.com/Articles/7933/Smart-Thread-Pool) for windows phone.
My problem is that...even for fetching around 80 items (in total) from 9 publications, it is taking more than a minute. How can i speed up the procedure?
Note: I have a two stage approach because many times the images are not available with headlines, and are only found in the article.
