Improve heavy work in a loop in multithreading
- by xjaphx
I have a little problem with my data processing.
public void ParseDetails()
{
for (int i = 0; i < mListAppInfo.Count; ++i)
{
ParseOneDetail(i);
}
}
For 300 records, it usually takes around 13-15 minutes.
I've tried to improve by using Parallel.For() but it always stop at some point.
public void ParseDetails()
{
Parallel.For(0, mListAppInfo.Count, i => ParseOneDetail(i));
}
In method ParseOneDetail(int index), I set an output log for tracking the record id which is under processing.
Always hang at some point, I don't know why..
ParseOneDetail(): 89 ...
ParseOneDetail(): 90 ...
ParseOneDetail(): 243 ...
ParseOneDetail(): 92 ...
ParseOneDetail(): 244 ...
ParseOneDetail(): 93 ...
ParseOneDetail(): 245 ...
ParseOneDetail(): 247 ...
ParseOneDetail(): 94 ...
ParseOneDetail(): 248 ...
ParseOneDetail(): 95 ...
ParseOneDetail(): 99 ...
ParseOneDetail(): 249 ...
ParseOneDetail(): 100 ...
_ <hang at this point>
Appreciate your help and suggestions to improve this.
Thank you!
Edit 1: update for method:
private void ParseOneDetail(int index)
{
Console.WriteLine("ParseOneDetail(): " + index + " ... ");
ApplicationInfo appInfo = mListAppInfo[index];
var htmlWeb = new HtmlWeb();
var document = htmlWeb.Load(appInfo.AppAnnieURL);
// get first one only
HtmlNode nodeStoreURL = document.DocumentNode.SelectSingleNode(Constants.XPATH_FIRST);
appInfo.StoreURL = nodeStoreURL.Attributes[Constants.HREF].Value;
}
Edit 2: This is the error output after a while running as Enigmativity suggest,
ParseOneDetail(): 234 ...
ParseOneDetail(): 87 ...
ParseOneDetail(): 235 ...
ParseOneDetail(): 236 ...
ParseOneDetail(): 88 ...
ParseOneDetail(): 238 ...
ParseOneDetail(): 89 ...
ParseOneDetail(): 90 ...
ParseOneDetail(): 239 ...
ParseOneDetail(): 92 ...
Unhandled Exception: System.AggregateException: One or more errors occurred. ---
> System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetResponse()
at HtmlAgilityPack.HtmlWeb.Get(Uri uri, String method, String path, HtmlDocum
ent doc, IWebProxy proxy, ICredentials creds) in D:\Source\htmlagilitypack.new\T
runk\HtmlAgilityPack\HtmlWeb.cs:line 1355
at HtmlAgilityPack.HtmlWeb.LoadUrl(Uri uri, String method, WebProxy proxy, Ne
tworkCredential creds) in D:\Source\htmlagilitypack.new\Trunk\HtmlAgilityPack\Ht
mlWeb.cs:line 1479
at HtmlAgilityPack.HtmlWeb.Load(String url, String method) in D:\Source\htmla
gilitypack.new\Trunk\HtmlAgilityPack\HtmlWeb.cs:line 1103
at HtmlAgilityPack.HtmlWeb.Load(String url) in D:\Source\htmlagilitypack.new\
Trunk\HtmlAgilityPack\HtmlWeb.cs:line 1061
at SimpleChartParser.AppAnnieParser.ParseOneDetail(ApplicationInfo appInfo) i
n c:\users\nhn60\documents\visual studio 2010\Projects\FunToolPack\SimpleChartPa
rser\AppAnnieParser.cs:line 90
at SimpleChartParser.AppAnnieParser.<ParseDetails>b__0(ApplicationInfo ai) in
c:\users\nhn60\documents\visual studio 2010\Projects\FunToolPack\SimpleChartPar
ser\AppAnnieParser.cs:line 80
at System.Threading.Tasks.Parallel.<>c__DisplayClass21`2.<ForEachWorker>b__17
(Int32 i)
at System.Threading.Tasks.Parallel.<>c__DisplayClassf`1.<ForWorker>b__c()
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
at System.Threading.Tasks.Task.<>c__DisplayClass7.<ExecuteSelfReplicating>b__
6(Object )
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceled
Exceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationTo
ken cancellationToken)
at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int
32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWit
hState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](TSource[] ar
ray, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Act
ion`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEveryt
hing, Func`1 localInit, Action`1 localFinally)
at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`
1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState
, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithE
verything, Func`1 localInit, Action`1 localFinally)
at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 source, Act
ion`1 body)
at SimpleChartParser.AppAnnieParser.ParseDetails() in c:\users\nhn60\document
s\visual studio 2010\Projects\FunToolPack\SimpleChartParser\AppAnnieParser.cs:li
ne 80
at SimpleChartParser.Program.Main(String[] args) in c:\users\nhn60\documents\
visual studio 2010\Projects\FunToolPack\SimpleChartParser\Program.cs:line 15
