Sometimes, on node reboot, grub stops and expects user input

[Egor Kotko]

{"build_id": "2014-08-13_02-01-17", "ostf_sha": "d2a894d228c1f3c22595a77f04b1e00d09d8e463", "build_number": "430", "auth_required": true, "api": "1.0", "nailgun_sha": "8676362d130a8fcb4336129babb9aff710926621", "production": "docker", "fuelmain_sha": "3c7a8aa6e35d9e4ae36c8b2ae3ea39baec76a211", "astute_sha": "b52910642d6de941444901b0f20e95ebbcb2b2e9", "feature_groups": ["mirantis"], "release": "5.1", "fuellib_sha": "b0951014f45ffbc5c94d2e07689843da3084027f"}

Sometimes ceph-node freeze on the grub-step of choosing which OS should be booted(see the screen)

http://jenkins-product.srt.mirantis.net:8080/view/0_master_swarm/job/master_fuelmain.system_test.ubuntu.thread_1/133/testReport/%28root%29/check_ceph_partitions_after_reboot/check_ceph_partitions_after_reboot/

[Vladimir Kuklin (vkuklin)]

this looks like local environment issue. or may be someone presses a key and grub stops the timer

[Egor Kotko (ykotko)]

I have reproduced this bug locally. The bug is floating.
There are several conditions how to get it:
- OS: Ubuntu
- start virtual machine, on the stage of booting "Force shutdown" after start virtual machine, sometimes it is necessary to do several times (shutdown/start).
- after you will get grub menu (see the screen)

As Workaround:
1) Edit /etc/grub.d/00_header
2) Line 236: set timeout=0
3) update-grub
4) Reboot virtual machine.

[Andrew Woodward (xarses)]

I think this should be high, we should attempt to reproduce and prevent issue.

[Vladimir Kuklin (vkuklin)]

It more looks like tests issue as grub will not stop timing out unless there is a key sent to the inout. We need to try to reproduce the issue and move it to 6.0 if it is not frequently reproduced.

[Mike Scherbakov (mihgen)]

I believe we should change the title then to "Sometimes, on node reboot, grub stops and expects user input"

[Egor Kotko (ykotko)]

More exact steps to reproduce:
1) Virtual machine with deployed role (OS Ubuntu)
2) Shutdown (wait until it will be powered off)
3) Start
4) On the stage of booting bios/grub destroy virtual machine
5) Start
After you will get the screen with boot menu.

On the hardware machine it should be reproducible only via electricity power-off on the stage of boot or force power off via button. Should not be very often.

[Dennis Dmitriev (ddmitriev)]

Reproduced the same issue on CI test: Ubuntu thread_1, http://jenkins-product.srt.mirantis.net:8080/view/0_master_swarm/job/master_fuelmain.system_test.ubuntu.thread_1/164/

[Dennis Dmitriev (ddmitriev)]

Works fine for Ubuntu:

1) in '/etc/default/grub' add 'GRUB_RECORDFAIL_TIMEOUT':

...
GRUB_TIMEOUT=10
GRUB_RECORDFAIL_TIMEOUT=$GRUB_TIMEOUT
...

2) Update the grub config:

sudo update-grub2

[Andrey Sledzinskiy (asledzinskiy)]

the same issue was reproduced on 5.1.1 CI
http://jenkins-product.srt.mirantis.net:8080/view/5.1_swarm/job/5.1_fuelmain.system_test.ubuntu.thread_1/32/testReport/junit/%28root%29/check_ceph_partitions_after_reboot/check_ceph_partitions_after_reboot/

Logs are attached

[Andrey Sledzinskiy (asledzinskiy)]

Also was reproduced on 6.0 CI tests
logs are attached

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/133028

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/133283

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/133028
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=7c973398356baca4faf7e80b084d23ec636d4fd9
Submitter: Jenkins
Branch: master

commit 7c973398356baca4faf7e80b084d23ec636d4fd9
Author: Stanislav Makar <email address hidden>
Date: Thu Nov 6 13:12:07 2014 +0000

    Disable the waiting for user input if the last boot failed

    Add the option GRUB_RECORDFAIL_TIMEOUT to grub and
    set it to the same value as the GRUB_TIMEOUT

    Change-Id: Iff1a65f90ecc1ff41763ed6b1a614a8b27589327
    Closes-bug: #1356278

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/133283
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=b73e3b1d5c945d52ed5204140a70f0d693777506
Submitter: Jenkins
Branch: stable/5.1

commit b73e3b1d5c945d52ed5204140a70f0d693777506
Author: Stanislav Makar <email address hidden>
Date: Thu Nov 6 13:12:07 2014 +0000

    Disable the waiting for user input if the last boot failed

    Add the option GRUB_RECORDFAIL_TIMEOUT to grub and
    set it to the same value as the GRUB_TIMEOUT

    Change-Id: Iff1a65f90ecc1ff41763ed6b1a614a8b27589327
    Closes-bug: #1356278

[Dennis Dmitriev (ddmitriev)]

Fix released on Fuel 6.0:
api: '1.0'
astute_sha: 0085021fe327f6f910901b3ca55051b1df33a96e
auth_required: true
build_id: 2014-11-13_22-57-40
build_number: '106'
feature_groups:
- mirantis
fuellib_sha: 045f2f77f35b79b0321c92882f39203f315faabb
fuelmain_sha: 4614db0701a8add2f60b8fffdc72f106621cbce2
nailgun_sha: ba5604db9424319e002dd9b5deafe28d31036840
ostf_sha: 82465a94eed4eff1fc8d8e1f2fb7e9993c22f068
production: docker
release: '6.0'
release_versions:
  2014.2-6.0:
    VERSION:
      api: '1.0'
      astute_sha: 0085021fe327f6f910901b3ca55051b1df33a96e
      build_id: 2014-11-13_22-57-40
      build_number: '106'
      feature_groups:
      - mirantis
      fuellib_sha: 045f2f77f35b79b0321c92882f39203f315faabb
      fuelmain_sha: 4614db0701a8add2f60b8fffdc72f106621cbce2
      nailgun_sha: ba5604db9424319e002dd9b5deafe28d31036840
      ostf_sha: 82465a94eed4eff1fc8d8e1f2fb7e9993c22f068
      production: docker
      release: '6.0'

[Dennis Dmitriev (ddmitriev)]

Released for 5.1.1:

api: '1.0'
astute_sha: 702af3db6f5bca92525bc8322d7d5d7675ec857e
auth_required: true
build_id: 2014-11-14_08-51-33
build_number: '9'
feature_groups:
- mirantis
fuellib_sha: 938850f89f27364207e24fee0f3b823cebb51083
fuelmain_sha: e5e534ade6f3765a87feee3d44d39df68ae28f80
nailgun_sha: bbc9dfe78a0c33040dcd16de9a40a3491788719c
ostf_sha: 64cb59c681658a7a55cc2c09d079072a41beb346
production: docker
release: 5.1.1
release_versions:
  2014.1.1-5.1.1:
    VERSION:
      api: '1.0'
      astute_sha: 702af3db6f5bca92525bc8322d7d5d7675ec857e
      build_id: 2014-11-14_08-51-33
      build_number: '9'
      feature_groups:
      - mirantis
      fuellib_sha: 938850f89f27364207e24fee0f3b823cebb51083
      fuelmain_sha: e5e534ade6f3765a87feee3d44d39df68ae28f80
      nailgun_sha: bbc9dfe78a0c33040dcd16de9a40a3491788719c
      ostf_sha: 64cb59c681658a7a55cc2c09d079072a41beb346
      production: docker
      release: 5.1.1