[Ubuntu 14.04] Image based provisioning of Ubuntu failed with Boot sector signature not found

[Andrey Sledzinskiy]

{

    "ostf_sha": "6c046b69d29021524906109f18092363505ee222",
    "release_versions":

{

    "2014.2-6.1":

{

    "VERSION":

{

    "ostf_sha": "6c046b69d29021524906109f18092363505ee222",
    "api": "1.0",
    "nailgun_sha": "b970afec55d3dc9c0ae32368e870d8f988dff5fa",
    "production": "docker",
    "python-fuelclient_sha": "521c2491f7f04f31d8c85db68499cd193d4904e3",
    "astute_sha": "31bae60eff9e8a1f85b04025145cfbcc65b537d9",
    "feature_groups":

            [
                "experimental"
            ],
            "release": "6.1",
            "fuelmain_sha": "",
            "fuellib_sha": "f2f719bc07cf78110fbe93acaf55aeb25223c0da"
        }
    }

},
"auth_required": true,
"api": "1.0",
"nailgun_sha": "b970afec55d3dc9c0ae32368e870d8f988dff5fa",
"production": "docker",
"python-fuelclient_sha": "521c2491f7f04f31d8c85db68499cd193d4904e3",
"astute_sha": "31bae60eff9e8a1f85b04025145cfbcc65b537d9",
"feature_groups":

    [
        "experimental"
    ],
    "release": "6.1",
    "fuelmain_sha": "",
    "fuellib_sha": "f2f719bc07cf78110fbe93acaf55aeb25223c0da"

}

Steps:
1. Create next cluster - HA , Ubuntu, flat nova-network, ceph for volumes and images, 3 controller, 2 compute, 2 ceph nodes
2. Start deployment

Actual - provisioning of the node failed with Boot sector signature not found

Logs are attached

[Vladimir Kozhukalov (kozhukalov)]

Error screenshot.

[Vladimir Kozhukalov (kozhukalov)]

Tried to boot a node directly from a disk (w/o chain loader). The error is the same which means that we really had a problem during provisioning. Fuel Agent logs are clear and does not contain any errors. Tried to run the provision script manually and then just after it finished, I tried to mount /, /boot, /proc, /dev, /sys file systems and run grub-install manually against all the node hard drives and then rebooted. The node managed to boot successfully with the error "diskfilter writes are not supported" which appeared just after grub menu (hidden) and before splash. Took a look at the /var/log/remote/node-N.domain.tld/bootstrap/messages.log file and it looks like the issue appears when sysrq message (unconditional reboot) is sent to a failed node while it is being provisioned.

[Vladimir Kozhukalov (kozhukalov)]

The reason of this issue to appear is exactly the same as for https://bugs.launchpad.net/fuel/+bug/1394599. Cobbler runs reboot command and reports reboot succeeded, but astute manages to get the type of node OS via Mcollective (/etc/systemtype) and runs node erasing procedure for the nodes which type==bootstrap. Erasing procedure in turn kicks sysrq trigger which breaks the node.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/154904

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/155814

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vladimir Kozhukalov (<email address hidden>) on branch: master
Review: https://review.openstack.org/154904

[OpenStack Infra (hudson-openstack)]

Change abandoned by Aleksandr Gordeev (<email address hidden>) on branch: stable/6.0
Review: https://review.openstack.org/155814