I am struggling a little coming to grips with the OTP development model as I convert some code into an OTP app.
I am essentially making a web crawler and I just don't quite know where to put the code that does the actual work.
I have a supervisor which starts my worker:
-behaviour(supervisor).
-define(CHILD(I, Type), {I, {I, start_link, []}, permanent, 5000, Type, [I]}).
init(_Args) ->
Children = [
?CHILD(crawler, worker)
],
RestartStrategy = {one_for_one, 0, 1},
{ok, {RestartStrategy, Children}}.
In this design, the Crawler Worker is then responsible for doing the actual work:
-behaviour(gen_server).
start_link() ->
gen_server:start_link(?MODULE, [], []).
init([]) ->
inets:start(),
httpc:set_options([{verbose_mode,true}]),
% gen_server:cast(?MODULE, crawl),
% ok = do_crawl(),
{ok, #state{}}.
do_crawl() ->
% crawl!
ok.
handle_cast(crawl}, State) ->
ok = do_crawl(),
{noreply, State};
do_crawl spawns a fairly large number of processes and requests that handle the work of crawling via http.
Question, ultimately is: where should the actual crawl happen? As can be seen above I have been experimenting with different ways of triggering the actual work, but still missing some concept essential for grokering the way things fit together.
Note: some of the OTP plumbing is left out for brevity - the plumbing is all there and the system all hangs together
