MOS 9.0 Install Fails From Virtual Media

[Russell Holloway]

Detailed Bug Description:

MOS 9.0 fails install when running post-installation scripts. Trying to install using virtual media over Dell iDrac (mounts media at /dev/sr1). It hangs at output "Running post-installation scripts", with the following error in log (screenshot attached):

Error code 127 running the kickstart script at line 1.

The last program log says "Running ... /bin/sh /tmp/ks-script-Lu5oNV" and I see that file in /mnt/sysimage/tmp but not in /tmp (I'm not sure which is supposed to have it, they differ and they both have ks-script-* files).

It looks like it is failing somewhere in anoconda-post-before-chroot, as that has log entries but anoconda-post-configure-repos.log is completely empty.

Steps to reproduce:

Attempt basic installation of MOS 9.0 using virtual media (Dell server in this case)

Expected Results:

Installation completes

Actual Results:

Installation never finishes post-installation scripts

Reproducibility:

Consistently for me

Workaround:

I'm not sure yet.

Impact:

Cannot install MOS 9.0

[Russell Holloway (russell-holloway)]

I'm actually having similar issue using MOS 8.0 as well.

[Alexander Kislitsky (akislitsky)]

@Russell, seems that it is an hardware specific issue. Could you please share the detailed information about used HW.

This is really strange situation: trying to execute /bin/bash at line 384 in anaconda-ks.cfg and have a 127 error. Thus /bin/bash is not found in PATH. May be virtual media is unmounted at this moment?

[Russell Holloway (russell-holloway)]

It's a Dell R230 using iDrac enterprise 8 to mount media.

So, after letting MOS 8.0 sit all night after "hanging", it actually still finished install - it just took a very long time (and it still had the same error show in logs). So I went back and let MOS 9.0 sit for longer, it appeared to hang on one of the ks-scripts, but then eventually went on and appears to have finished (haven't verified completely since I'm pending some routing changes for external uplink before finishing fuelmenu setup).

So, this *could* be related to https://bugs.launchpad.net/fuel/+bug/1564350 but I'm not sure. It just takes a long time, and I don't have external internet available during post-installation script yet. The errors I see are different, but symptoms similar I suppose.

[Roman Vyalov (r0mikiam)]

Reassign to Fuel team

[Vladimir Kozhukalov (kozhukalov)]

This bug is definitely not the same with https://bugs.launchpad.net/fuel/+bug/1564350. This error is related to script parsing. All our post install scripts start with shebang '#!/bin/bash' and for some reason anaconda can not parse this script. The error occurs when trying to run this script https://github.com/openstack/fuel-main/blob/stable/mitaka/iso/ks.template#L379 which is to be run in chroot /mnt/sysimage. When running in chroot this /mnt/sysimage/tmp should be available as /tmp. So the issue could be related to the broken content of the file /mnt/sysimage/tmp/ks-script-Lu5oNV or inability to chroot due to some issues about sysfs procfs tmpfs, etc.

Most likely, the issue could be solved by tuning virtual drive /dev/sr1 parameters like timeouts, keepalive parameters, etc.

[Vladimir Kozhukalov (kozhukalov)]

@russel, could you please tune virtual drive parameters and if it does not help, please append anaconda logs to the bug?

[Bruce Basil Mathews (bmathews-l)]

DTV installing MOS 9.0 Fuel on Dell r630. We are still getting an error as it is looking for a kickstart file on a CDRM regardless of whether we boot from a USB or an ISO... the bios settings still indicate 'cdrom' versus some other mount point...

[Vladimir Kozhukalov (kozhukalov)]

@Bruce, could you please attach anaconda logs to the bug?

[Vladimir Kozhukalov (kozhukalov)]

Valueable comment from Stanislaw Bogatkin:

"I believe that problem was introduced in 9114053378be00cd4d94d849ffc07903f407153e commit and it is a bug right now. We shouldn't use inst.repo=cdrom:LABEL='blah' for installation from usb stick, cause usb stick have different way to load than cdrom one. inst.repo=hd:device:/path and inst.ks=hd:device:/path should be used instead as it was in previous versions of Fuel and as it said in official documentation [0]. Feel free to fill a bug about it if you think similar way and assign it to me."

[0] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Installation_Guide/chap-anaconda-boot-options.html

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/350660

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/350660
Committed: https://git.openstack.org/cgit/openstack/fuel-main/commit/?id=08f1283db6c2dad75e2320b09446ca8a8c3898aa
Submitter: Jenkins
Branch: master

commit 08f1283db6c2dad75e2320b09446ca8a8c3898aa
Author: Vladimir Kozhukalov <email address hidden>
Date: Wed Aug 3 18:39:05 2016 +0300

    Change EFI fs label

    If EFI partition does not have specific label
    different from ISO label, then udev first creates
    /dev/disk/by-label for 9660 file system and then
    overrides this link with the link pointing to
    EFI partition.

    Change-Id: Id841c5b306940fd157aee527ad69b0995bcf9cd8
    Closes-Bug: #1605740

[Margarita Bliznikova (mbliznikova)]

Parameter that allows correctly load the kickstart from usb stick was specified.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-main 10.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-main 10.0.0 release.