Terasort (hadoop 2.7.1) failed on Ubuntu 1604
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
hadoop (Ubuntu) |
Confirmed
|
Undecided
|
Unassigned |
Bug Description
When I run terasort with Hadoop 2.7.1 on Ubuntu 1604, in 3 slaves and 1 master, with 500000000 records, in the middle of mapreduce job, some Ubuntu slave nodes will not able to be connected.
In this case, we are not able to create ssh connection to those slave nodes (connection refused).
If we login the slave node, then we will find:
1. dmesg shows systemd-journald received SIGTERM;
2. Several errors are found from /var/log/syslog. Iscsid reports semop down failed 22.
This is the Terasort output:
16/06/10 03:39:25 INFO terasort.TeraSort: starting
16/06/10 03:39:27 INFO input.FileInput
Spent 336ms computing base-splits.
Spent 9ms computing TeraScheduler splits.
Computing input splits took 348ms
Sampling 10 splits of 38
Making 7 from 100000 sampled records
Computing parititions took 1396ms
Spent 1749ms computing partitions.
16/06/10 03:39:29 INFO client.RMProxy: Connecting to ResourceManager at master/
16/06/10 03:39:30 INFO mapreduce.
16/06/10 03:39:30 INFO Configuration.
16/06/10 03:39:30 INFO mapreduce.
16/06/10 03:39:30 INFO impl.YarnClient
16/06/10 03:39:30 INFO mapreduce.Job: The url to track the job: http://
16/06/10 03:39:30 INFO mapreduce.Job: Running job: job_14655549434
16/06/10 03:40:03 INFO mapreduce.Job: Job job_14655549434
16/06/10 03:40:03 INFO mapreduce.Job: map 0% reduce 0%
16/06/10 03:44:54 INFO mapreduce.Job: map 1% reduce 0%
16/06/10 03:45:12 INFO mapreduce.Job: map 2% reduce 0%
..................
16/05/25 00:35:53 INFO mapreduce.Job: map 69% reduce 0%
16/05/25 00:35:54 INFO mapreduce.Job: map 75% reduce 0%
16/05/25 00:35:56 INFO mapreduce.Job: map 88% reduce 0%
16/05/25 00:35:57 INFO ipc.Client: Retrying connect to server: ubuntubm10/
affects: | ubuntu → hadoop (Ubuntu) |
Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https:/ /wiki.ubuntu. com/Bugs/ FindRightPackag e. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.
To change the source package that this bug is filed about visit https:/ /bugs.launchpad .net/ubuntu/ +bug/1594534/ +editstatus and add the package name in the text box next to the word Package.
[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]