Hadoop reduce task gets hung
- by user806098
I set up a hadoop cluster with 4 nodes, When running a map-reduce task, the map task finishes quickly, while the reduce task hangs at 27% percent. I checked the log, it's that the reduce task fails to fetch map output from map nodes.
The job tracker log of master shows messages like this:
2011-06-27 19:55:14,748 INFO org.apache.hadoop.mapred.JobTracker: Adding task (REDUCE) 'attempt_201106271953_0001_r_000000_0' to tip task_201106271953_0001_r_000000, for tracker 'tracker_web30.bbn.com.cn:localhost/127.0.0.1:56476'
And the name node log of master shows messages like this:
2011-06-27 14:00:52,898 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 54310, call register(DatanodeRegistration(202.106.199.39:50010, storageID=DS-1989397900-202.106.199.39-50010-1308723051262, infoPort=50075, ipcPort=50020)) from 192.168.225.19:16129: error: java.io.IOException: verifyNodeRegistration: unknown datanode 202.106.199.3 9:50010
However, neither the "web30.bbn.com.cn" or 202.106.199.39, 202.106.199.3 is the slave node. I think such ip/domains appear because hadoop fails to resolve a node(first in the Intranet DNS server), then it goes to a higher-level DNS server, later to the top, still fails, then the "junk" ip/domains are returned.
But I checked my config, it goes like this:
/etc/hosts:
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
192.168.225.16 master
192.168.225.66 slave1
192.168.225.20 slave5
192.168.225.17 slave17
conf/core-site.xml:
hadoop.tmp.dir
/root/hadoop_tmp/hadoop_${user.name}
fs.default.name
hdfs://master:54310
io.sort.mb
1024
hdfs-site.xml:
dfs.replication
3
masters:
master
slaves:
master
slave1
slave5
slave17
Also, all firewalls(iptables) are turned off, and ssh between each 2 nodes is ok.
so I don't know where exact the error comes from. Please help. Thanks a lot.
