I have an application which downloads more than 4500 html pages from 62 target hosts using HttpClient (4.1.3 or 4.2-beta). It runs on Windows 7 64-bit. Processor - Core i7 2600K. Network bandwidth - 54 Mb/s.
At this moment it uses such parameters:
DefaultHttpClient and PoolingClientConnectionManager;
Also it hasIdleConnectionMonitorThread from
http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html;
Maximum total connections = 80;
Default maximum connections per route = 5;
For thread management it uses ForkJoinPool with the parallelism
level = 5 (Do I understand correctly that it is a number of working
threads?)
In this case my network usage (in Windows task manager) does not rise above 2.5%. To download 4500 pages it takes 70 minutes. And in HttpClient logs I have such things:
DEBUG ForkJoinPool-2-worker-1
[org.apache.http.impl.conn.PoolingClientConnectionManager]: Connection
released: [id: 209][route: {}-http://stackoverflow.com][total kept
alive: 6; route allocated: 1 of 5; total allocated: 10 of 80]
Total allocated connections do not raise above 10-12, in spite of that I've set it up to 80 connections.
If I'll try to rise parallelism level to 20 or 80, network usage remains the same but a lot connection time-outs will be generated.
I've read tutorials on hc.apache.org (HttpClient Performance Optimization Guide and HttpClient Threading Guide) but they does not help.
Task's code looks like this:
public class ContentDownloader extends RecursiveAction {
private final HttpClient httpClient;
private final HttpContext context;
private List<Entry> entries;
public ContentDownloader(HttpClient httpClient, List<Entry> entries){
this.httpClient = httpClient;
context = new BasicHttpContext();
this.entries = entries;
}
private void computeDirectly(Entry entry){
final HttpGet get = new HttpGet(entry.getLink());
try {
HttpResponse response = httpClient.execute(get, context);
int statusCode = response.getStatusLine().getStatusCode();
if ( (statusCode >= 400) && (statusCode <= 600) ) {
logger.error("Couldn't get content from " + get.getURI().toString() + "\n" + response.toString());
} else {
HttpEntity entity = response.getEntity();
if (entity != null) {
String htmlContent = EntityUtils.toString(entity).trim();
entry.setHtml(htmlContent);
EntityUtils.consumeQuietly(entity);
}
}
} catch (Exception e) {
} finally {
get.releaseConnection();
}
}
@Override
protected void compute() {
if (entries.size() <= 1){
computeDirectly(entries.get(0));
return;
}
int split = entries.size() / 2;
invokeAll(new ContentDownloader(httpClient, entries.subList(0, split)),
new ContentDownloader(httpClient, entries.subList(split, entries.size())));
}
}
And the question is - what is the best practice to use multi threaded HttpClient, may be there is a some rules for setting up ConnectionManager and HttpClient? How can I use all of 80 connections and raise network usage?
If necessary, I will provide more code.
