Ubuntu Provisioning Failure

Bug #1319995 reported by Dmitry Ilyin on 2014-05-15
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Vladimir Kozhukalov
4.1.x
High
Fuel Library (Deprecated)

Bug Description

Ubuntu provisioning fails most of the time.

ISO: 206 (started from 19?)

Ubuntu HA Neutron-GRE
Ceph for both CInder and Glance

2 x Controller + Ceph OSD
1 x Compute

2014-05-15 19:43:09 NOTICE
 Unexpected error; command not executed: 'hwclock --systohc'
2014-05-15 19:43:09 NOTICE
 Unexpected error; command not executed: 'sed -i s/msg_age = Time.now.utc.to_i - msgtime/msg_age = 0 #Time.now.utc.to_i - msgtime/g /usr/lib/ruby/1.8/mcollective/message.rb'
2014-05-15 19:43:09 NOTICE
 Unexpected error; command not executed: 'sh -c echo ${0} | base64 --decode | gunzip -c | sh - H4sIAMwXdVMC/5WTX2vCMBTF3/spAn0b2CZV6VZQEHVsD9uD+C4xvbXBmkia6tdf0jj/lUpLoYHcc84vvTf1fR8xKTK+qxQXO1SlcEKqKqBEvu97wHKJpigEzUJbCutSkIYxHhxBlbzUIPRAgA7qivczW8+/ZovFajKOkvEowThJcbIlyTBD37/r5epzNl9OQOcRCgu+daFnxTVsTMimNYTGyZAkH6PHENIvJH63b0weQ/CLEM9/6g8XGlRG2X1/SlkpBq5JxnqWah/edKZZb+i/h82yd9inXLWbHYJWWqLCjGL6AsMzlu0GRuU83O5bEzdqs8rjlrJ9t4jL4LtLHayea41Lc3bs4reOzrCb+IojvXGkD4484XBvHO6Dw80LV7
2014-05-15 19:43:07 NOTICE
 Unexpected error; command not executed: 'update-grub'
2014-05-15 19:43:07 NOTICE
 Unexpected error; command not executed: 'grub-install /dev/vda'
2014-05-15 19:43:07 NOTICE
 Unexpected error; command not executed: 'grub-mkdevicemap'
2014-05-15 19:43:07 NOTICE
 Unexpected error; command not executed: 'grub-mkconfig'
2014-05-15 19:43:07 NOTICE
 Unexpected error; command not executed: 'sed -i -e s/.*GRUB_TERMINAL.*/GRUB_TERMINAL=console/g -e s/.*GRUB_GFXMODE.*/#GRUB_GFXMODE=640x480/g -e s/.*GRUB_CMDLINE_LINUX.*/GRUB_CMDLINE_LINUX=" console=ttyS0,9600 console=tty0 rootdelay=90 nomodeset "/g /etc/default/grub'
2014-05-15 19:43:07 NOTICE
 Unexpected error; command not executed: 'apt-get install -y grub-pc'
2014-05-15 19:43:07 NOTICE
 Unexpected error; command not executed: 'sgdisk --typecode=6:4fbd7e29-9d25-41b8-afd0-062c0ceff05d /dev/vda'
2014-05-15 19:42:38 NOTICE

2014-05-15 19:42:38 NOTICE
 Unable to locate package language-support-en
2014-05-15 19:42:38 NOTICE
 :
2014-05-15 19:42:38 NOTICE
 E
2014-05-15 19:42:38 NOTICE

2014-05-15 19:42:38 NOTICE
 Reading state information...

On the node's log:

May 15 19:38:51 main-menu[413]: DEBUG: resolver (libc6-udeb): package doesn't exist (ignored)
May 15 19:38:51 main-menu[413]: DEBUG: resolver (libnewt0.52): package doesn't exist (ignored)
May 15 19:38:51 main-menu[413]: INFO: Menu item 'network-preseed' selected
May 15 19:38:55 preseed: successfully loaded preseed file from http://10.20.0.2/cblr/svc/op/ks/system/node-1
May 15 19:38:55 preseed: running preseed command preseed/early_command: /bin/mkdir /.ssh ; /bin/echo "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAx1gd5aphXBfaRaz8bwQLY1EfDkLldMPBJRxvwf1BapDNnE5olKjZhJS8k5mjBHlu8sEf6laof2ixoW95480cnPwGaJfTrEGgEIA2/GRTheJ8vFsD4JfipXuWUISIWVMKu2nYKXcCpWLdGZQUV9uTcJn9HDeXSLDG0/hCsub41ruOW9hbOx0/lxY5NNb1ybi/MfVTZUZjgSdPM7M9+o7MXKj29ZNyuDo+ZqhpkILTnU3ru3vmid0C2UgLiJvIs7F88eeO71ssjuYJOpqAh2U+rPUmt0K1StUlvLyYOR/+ZzwMcgQNCgCgLLnue8axj7it7kCcJnExg8yJhWKCG7EKjQ== <email address hidden>" >
May 15 19:38:55 log-output: sed: /etc/ssh/sshd_config: No such file or directory
May 15 19:38:55 log-output: killall: sshd: no process killed

SSH to node is working. Key is present. sshd_config is present too.

# cat /etc/ssh/sshd_config
# Installer generated configuration file

Port 22
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
LoginGraceTime 600
PermitRootLogin yes
StrictModes yes
PermitEmptyPasswords no
PasswordAuthentication yes
UsePrivilegeSeparation no

X11Forwarding no
PrintMotd yes
TCPKeepAlive yes

Did this command fail?
./iso/ks.template:sed -i -e "/^\s*GSSAPICleanupCredentials yes/d" -e "/^\s*GSSAPIAuthentication yes/d" /etc/ssh/sshd_config
No. Looks like it was successful... What happened then?

Dmitry Ilyin (idv1985) wrote :
Mike Scherbakov (mihgen) on 2014-05-15
Changed in fuel:
assignee: nobody → Fuel Library Team (fuel-library)
Dmitry Ilyin (idv1985) wrote :

Ubuntu HA without Ceph - provisioning and deployment went normally.

Dmitry Ilyin (idv1985) wrote :

Deployment with external ceph osd nodes went successful too. This pretty much narrows the problem down to situations when ceph osd is placed on controller.

Dmitry Borodaenko (angdraug) wrote :

Data point: Ubuntu with ceph-osd on controller works for me with ISO #192.

Dmitry Ilyin (idv1985) wrote :

--- pmanager-new.py 2014-05-19 19:13:44.630723562 +0000
+++ pmanager-old.py 2014-05-18 21:24:28.000000000 +0000
@@ -104,7 +104,7 @@

     def _gettabfsoptions(self, vol):
         if self._gettabfstype(vol) == "xfs":
- return "-f"
+ return "-f -s size=512"
         return ""

     def _getfstype(self, vol):
@@ -277,13 +277,14 @@
                               "--typecode={0}:{1} {2}".format(
                                   pcount, part["partition_guid"],
                                   self._disk_dev(disk)))
- if size > 0 and size <= 16777216 and part["mount"] != "none":
+ if size > 0 and size <= 16777216 and tabfstype != "xfs":
                     self.kick("partition {0} "
                               "--onpart={2}"
                               "{3}{4}".format(part["mount"], size,
                                            self._disk_dev(disk),
                                            self._pseparator(disk["id"]),
                                            pcount))
+
                 else:
                     if part["mount"] != "swap" and tabfstype != "none":
                         disk_label = self._getlabel(part.get('disk_label'))
@@ -410,7 +411,7 @@
                 tabfstype = self._gettabfstype(lv)
                 tabfsoptions = self._gettabfsoptions(lv)

- if size > 0 and size <= 16777216:
+ if size > 0 and size <= 16777216 and tabfstype != "xfs":
                     self.kick("logvol {0} --vgname={1} --size={2} "
                               "--name={3} {4}".format(
                                   lv["mount"], vg["id"], size,
@@ -587,7 +588,7 @@

     def _fsoptions(self, fstype):
         if fstype == "xfs":
- return "-f"
+ return "-f -s size=512"
         return ""

     def _umount_target(self):
@@ -616,18 +617,14 @@
         """This method puts script which splits
         long line and sends them to logger
         #!/bin/sh
-
- read string
         chunk=80
-
- iterations=`expr ${#string} / $chunk + 1`
- i=0
- while [ $i -le $(( iterations - 1)) ]; do
- start_sym=$(( $i * ${chunk} + 1 ))
- end_sym=$(( $(( $i + 1 )) * ${chunk}))
- echo $string | cut -c ${start_sym}-${end_sym} | logger
- i=$(( i + 1 ))
- done
+ while read string; do
+ iterations=`expr ${#string} / $chunk + 1`; i=0;
+ while [ $i -le $(( iterations - 1)) ]; do
+ start_sym=$(( $i * ${chunk} + 1 ))
+ end_sym=$(( $(( $i + 1 )) * ${chunk}))
+ echo $string | cut -c ${start_sym}-${end_sym} | logger
+ i=$(( i + 1 )); done; done;
         """
         return (
             """echo '#!/bin/sh' > /tmp/long_logger.sh;""",

This is diff between the working version of pmanager.py and the current one.

Changed in fuel:
status: New → Triaged
Mike Scherbakov (mihgen) on 2014-05-20
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Dmitry Ilyin (idv1985)
Mike Scherbakov (mihgen) wrote :

Moved this to High, as it fails only in very certain cases.

Changed in fuel:
importance: Critical → High

Fix proposed to branch: master
Review: https://review.openstack.org/94619

Changed in fuel:
status: Triaged → In Progress
Changed in fuel:
assignee: Dmitry Ilyin (idv1985) → Vladimir Kozhukalov (kozhukalov)

Reviewed: https://review.openstack.org/94619
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=347646bd46052e1db938b0c4f8464eab44e247a1
Submitter: Jenkins
Branch: master

commit 347646bd46052e1db938b0c4f8464eab44e247a1
Author: Dmitry Ilyin <email address hidden>
Date: Wed May 21 19:51:27 2014 +0400

    Fix ubuntu provisioning

    This patch fixes partitioning counting. We use kind of
    '$(readlink -f $(ls /dev/disk/by-path/foo) )' as a
    key for a partitioning counter, but in that particular place
    it was like /dev/disk/by-path/foo.

    Change-Id: Iafbfcda96d0904ff87781df4d3b04d9293656d04
    Closes-Bug: 1319995

Changed in fuel:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/96859
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=e6d2ddaaff69cd12f18724c000d0265823bae019
Submitter: Jenkins
Branch: stable/4.1

commit e6d2ddaaff69cd12f18724c000d0265823bae019
Author: Dmitry Ilyin <email address hidden>
Date: Wed May 21 19:51:27 2014 +0400

    Fix ubuntu provisioning

    This patch fixes partitioning counting. We use kind of
    '$(readlink -f $(ls /dev/disk/by-path/foo) )' as a
    key for a partitioning counter, but in that particular place
    it was like /dev/disk/by-path/foo.

    Change-Id: Iafbfcda96d0904ff87781df4d3b04d9293656d04
    Closes-Bug: 1319995

Meg McRoberts (dreidellhasa) wrote :

Marked as fixed in 4.1.1 Release Notes

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments