The file sahara-image-elements/elements/hadoop/post-install.d/firstboot is installed on images as /etc/rc.d/rc.local. It contains lines to modify /mnt:
chown root:root /mnt
mkdir -p /mnt/log/hadoop
chown hadoop:hadoop /mnt/log/hadoop
The vanilla 1.2.1 plugin also modifies /mnt in the /tmp/sahara-hadoop-init.sh script created by the plugin:
chown -R hadoop:hadoop /mnt
chmod -R 755 /mnt
When the rc.local file is run by systemd as the rc-local.service, it creates a race condition with the /tmp/sahara-hadoop-init.sh script. (timestamps showing the race in a failed case are shown below). In the failure case, the datanode service on a worker will not start because /mnt has the wrong ownership (https://bugs.launchpad.net/sahara/+bug/1295807)
This is particularly evident with Fedora 1.2.1 images since the rc.local file contains a 'sleep' statement and a "until do" loop in the Fedora case (but it could theoretically happen with any plugin that modifies /mnt or other things that firstboot touches)
Fedora )
sleep 20
rm /etc/resolv.conf
service network restart
if [ $(lsb_release -rs) -ge '19' ]; then
user=fedora
else
user=ec2-user
fi
until [[ -n $(grep "$user:" /etc/passwd) && -n $(cat /etc/group | grep "$user:") ]];
do
sleep 1
done
chown -R $user:$user /home/$user
;;
Below is output from a worker node where a timestamp was added to the /etc/rc.d/rc.local file on the image using virt-edit, and another timestamp was added to the /tmp/sahara-hadoop-init.sh script created by the plugin code:
[root@testcluster-workers-003 tmp]# cat rc-local.debug
chowned mnt at Thu May 29 14:24:45 UTC 2014
[root@testcluster-workers-003 tmp]# cat sahara-hadoop-init.log
+ echo -n
+ cat /etc/hadoop/hadoop-env.sh
+ cp /tmp/hadoop-env.sh /etc/hadoop/hadoop-env.sh
+ sed -i 's,export HADOOP_LOG_DIR=.*,export HADOOP_LOG_DIR=/mnt/log/hadoop/$USER/,' /etc/hadoop/hadoop-env.sh
+ sed -i 's,export HADOOP_SECURE_DN_LOG_DIR=.*,export HADOOP_SECURE_DN_LOG_DIR=/mnt/log/hadoop/hdfs,' /etc/hadoop/hadoop-env.sh
+ chown -R hadoop:hadoop /mnt
+ chmod -R 755 /mnt
++ date
+ echo 'sahara-init finished at' Thu May 29 14:24:36 UTC 2014
sahara-init finished at Thu May 29 14:24:36 UTC 2014
We have a machine where this is very easy to reproduce, probably 50% of the time or greater using Fedora images. However, it does not appear on all machines (the nature of a race)
Note, made https:/