[DIB] Use of rc.local to modify paths creates race condition
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Sahara |
Fix Released
|
High
|
Ivan Berezovskiy |
Bug Description
The file sahara-
chown root:root /mnt
mkdir -p /mnt/log/hadoop
chown hadoop:hadoop /mnt/log/hadoop
The vanilla 1.2.1 plugin also modifies /mnt in the /tmp/sahara-
chown -R hadoop:hadoop /mnt
chmod -R 755 /mnt
When the rc.local file is run by systemd as the rc-local.service, it creates a race condition with the /tmp/sahara-
This is particularly evident with Fedora 1.2.1 images since the rc.local file contains a 'sleep' statement and a "until do" loop in the Fedora case (but it could theoretically happen with any plugin that modifies /mnt or other things that firstboot touches)
Fedora )
sleep 20
rm /etc/resolv.conf
service network restart
if [ $(lsb_release -rs) -ge '19' ]; then
else
fi
until [[ -n $(grep "$user:" /etc/passwd) && -n $(cat /etc/group | grep "$user:") ]];
do
sleep 1
done
chown -R $user:$user /home/$user
;;
Below is output from a worker node where a timestamp was added to the /etc/rc.d/rc.local file on the image using virt-edit, and another timestamp was added to the /tmp/sahara-
[root@testclust
chowned mnt at Thu May 29 14:24:45 UTC 2014
[root@testclust
+ echo -n
+ cat /etc/hadoop/
+ cp /tmp/hadoop-env.sh /etc/hadoop/
+ sed -i 's,export HADOOP_
+ sed -i 's,export HADOOP_
+ chown -R hadoop:hadoop /mnt
+ chmod -R 755 /mnt
++ date
+ echo 'sahara-init finished at' Thu May 29 14:24:36 UTC 2014
sahara-init finished at Thu May 29 14:24:36 UTC 2014
We have a machine where this is very easy to reproduce, probably 50% of the time or greater using Fedora images. However, it does not appear on all machines (the nature of a race)
Changed in sahara: | |
milestone: | juno-1 → juno-2 |
Changed in sahara: | |
milestone: | juno-2 → juno-3 |
Changed in sahara: | |
milestone: | juno-3 → next |
Changed in sahara: | |
milestone: | next → 2014.2 |
assignee: | nobody → Ivan Berezovskiy (iberezovskiy) |
tags: | added: dib |
Note, made https:/ /bugs.launchpad .net/sahara/ +bug/1295807 a duplicate of this new bug