While scheduling and resource allocation control has been present in Hadoop since 0.20, a lot of people haven't
discovered or utilized it in their initial investigations of
the Hadoop ecosystem. We could chalk this up to many things:
Organizations are still determining what their dataflow and analysis workloads will comprise
Small deployments under tests aren't likely to show
the signs
of strains that would send someone looking for resource allocation
options
The default scheduling options --
the FairScheduler and
the
CapacityScheduler -- are not placed in
the most prominent position
within
the Hadoop documentation.
However, for production deployments, it's wise to start with at
least
the foundations of scheduling in place so that you can tune
the cluster as workloads emerge. To do that, we have to ask ourselves
something about what
the off-the-rack scheduling options are. We have
some choices:
The FairScheduler, which will work to ensure resource allocations are enforced on a per-job basis.
The CapacityScheduler, which will ensure resource allocations are enforced on a per-queue basis.
Writing your own implementation of
the abstract class
org.apache.hadoop.mapred.job.TaskScheduler is an option, but usually
overkill.
If you're going to have several concurrent users and leverage
the more
interactive aspects of
the Hadoop environment
(e.g. Pig and Hive scripting),
the FairScheduler is definitely
the way
to go.
In particular, we can do user-specific pools so that default users get
their fair share, and specific users are given
the resources their
workloads require.
To enable fair scheduling, we're going to need to do a couple of things.
First, we need to tell
the JobTracker that we want to use scheduling and where we're going to be defining our allocations.
We do this by adding
the following to
the
mapred-site.xml file in HADOOP_HOME/conf:
<property>
<name>mapred.jobtracker.taskScheduler</name>
<value>org.apache.hadoop.mapred.FairScheduler</value>
</property>
<property>
<name>mapred.fairscheduler.allocation.file</name>
<value>/path/to/allocations.xml</value>
</property>
<property>
<name>mapred.fairscheduler.poolnameproperty</name>
<value>pool.name</value>
</property>
<property>
<name>pool.name</name>
<value>${user.name}</name>
</property>
What we've done here is simply tell
the JobTracker that we'd like to
task scheduling to use
the FairScheduler class rather than a single FIFO
queue.
Moreover, we're going to be defining our resource pools and
allocations in a file called allocations.xml
For reference,
the allocation file is read every 15s or so, which
allows for tuning allocations without having to take down
the
JobTracker.
Our allocation file is now going to look a little like this
<?xml version="1.0"?>
<allocations>
<pool name="dan">
<minMaps>5</minMaps>
<minReduces>5</minReduces>
<maxMaps>25</maxMaps>
<maxReduces>25</maxReduces>
<minSharePreemptionTimeout>300</minSharePreemptionTimeout>
</pool>
<mapreduce.job.user.name="dan">
<maxRunningJobs>6</maxRunningJobs>
</user>
<userMaxJobsDefault>3</userMaxJobsDefault>
<fairSharePreemptionTimeout>600</fairSharePreemptionTimeout>
</allocations>
In this case, I've explicitly set my username to have upper and lower
bounds on
the maps and reduces, and allotted myself double
the number of
running jobs.
Now, if I run hive or pig jobs from either
the console or via
the
Hue web interface, I'll be treated "fairly" by
the JobTracker.
There's a lot more tweaking that can be done to
the allocations
file, so it's best to dig down into
the
description
and start trying out allocations that might fit your workload.
