Master node setup failed due to kickstart doesn't loaded

[Alexey. Kalashnikov]

Swarm test Create environment and set up master node - thread_3 failed on step
"Create environment and set up master node"
with error:
http://paste.openstack.org/show/557570/

console output:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.thread_3/28/console

seems like kickstart script doesn't loaded properly.

trying to reverting snapshot leaded to error:
http://paste.openstack.org/show/557573/

[Tatyana Kuterina (tkuterina)]

This issue is reproduced on CI 9.1 snapshot #140:
Test: Create environment and set up master node - command_line
Test Group: setup_master_command_line
Trace: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.command_line/29/testReport/(root)/setup_master/setup_master/

[Alexey. Kalashnikov (akalashnikov)]

this is all what i can get from env.

manually try to run test second time on that env, had passed

[Alexander Kurenyshev (akurenyshev)]

I want to add some clarification, how I get the error with kickstart.
I reverted the env with error and connected to the master's node VNC console.
Here I saw the error from loader that it couldn't load the kickstart file from Fuel_Openstack iso.
I just rebooted the VM and Fuel iso boot menu appeared. I re-wrote boot parameters such as networks and started boot from the iso. After that everything worked fine and installation have started.

[Dmitry Pyzhov (dpyzhov)]

Could you provide an environment for investigation?

[Nastya Urlapova (aurlapova)]

https://product-ci.infra.mirantis.net/view/9.x_swarm/job/9.x.system_test.ubuntu.ha_neutron_destructive/30/testReport/(root)/setup_master/setup_master/

[Alex Schultz (alex-schultz)]

Last failure for the setup_master seems to be a python exception in anaconda related to NetworkManager. This would point to either a CentOS issue or perhaps a fuel-qa libvirt/network issue.

[Dmitry Teselkin (teselkin-d)]

Please attach full logs from /tmp and /root from master node that failed.

[Alex Schultz (alex-schultz)]

In this case it would probably be more useful to login to the environment that is available. But attached are the /tmp and /root contents.

[Dmitry Teselkin (teselkin-d)]

Possible fix in upstream https://github.com/rhinstaller/anaconda/pull/746

[Dmitry Teselkin (teselkin-d)]

Applying a fix from upstream requires to rebuild LiveOS image [1] for which we have no sources, so the only way to do this is to download and unpack the image (git?), patch, and re-pack again before building an ISO. I'm afraid that this will break more than fix.

This bug quite hard to reproduce, and there is another bug with similar reproduction rate [2]. The solution for [2] (restart test once if master node failed to deploy) should work here too, so I suggest to use this approach.

[1] http://mirror.centos.org/centos/7/os/x86_64/LiveOS/
[2] https://bugs.launchpad.net/fuel/+bug/1587411

[Alexander Kurenyshev (akurenyshev)]

There were no reproduces since 2016-08-17, moved to Incomplete.
Please, feel free to reopen this bug with appropriate info, if it appears again.

[Alexander Kurenyshev (akurenyshev)]

UPD: sorry, moved to Invalid