Instance creation failed on Ubuntu, Neutron cluster with Unexpected error while running command.\nCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf blkid -o value -s TYPE /dev/nbd14

[Andrey Sledzinskiy]

Description - instance creation failed with Unexpected error while running command.\nCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf blkid -o value -s TYPE /dev/nbd14

Steps:
1. Create next cluster - Ubuntu, HA, Neutron Gre, 3 controller, 2 compute, 1 cinder
2. Deploy cluster
3. Create keypair for instance
. openrc; nova keypair-add instancekey > /root/.ssh/webserver_rsa
4. Try to create instance with keypair

Expected - instance was created
Actual - instance failed to be created

From node-1 nova-conductor.log:

2015-07-13 07:07:07 ERR nova.scheduler.utils [req-b302c9f9-5ab2-4447-9948-e1906e2c7144 3ee23604f8ce4cceb1b7f19f5d8e402c 80a4fa5ae88442c5af8b5525cd2b580f - - -] [instance: e96a177f-f54b-4f26-bac7-042234d79528] Error from last host: node-3.test.domain.local (node node-3.test.domain.local): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2219, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2362, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance e96a177f-f54b-4f26-bac7-042234d79528 was re-scheduled: Unexpected error while running command.\nCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf blkid -o value -s TYPE /dev/nbd14\nExit code: 2\nStdout: u''\nStderr: u''\n"]

After second try to create instance it was created without errors

{

    "build_id": "2015-07-12_15-52-44",
    "build_number": "31",
    "release_versions":

{

    "2014.2.2-7.0":

{

    "VERSION":

{

    "build_id": "2015-07-12_15-52-44",
    "build_number": "31",
    "api": "1.0",
    "fuel-library_sha": "49c7ddeb5e4257bb52862bc5aa22600df71bb52a",
    "nailgun_sha": "60f9bf536e30efd896b7b4da1830e71adda19e30",
    "feature_groups":

            [
                "mirantis"
            ],
            "openstack_version": "2014.2.2-7.0",
            "production": "docker",
            "python-fuelclient_sha": "accd6493bf034ba7c70c987ace8f1dcd960cbdf5",
            "astute_sha": "9cbb8ae5adbe6e758b24b3c1021aac1b662344e8",
            "fuel-ostf_sha": "62785c16f8399f30526d24c52bb9ca23e1585bfb",
            "release": "7.0",
            "fuelmain_sha": "28551be12a050acb9a633933ed6a8b25e2dc411c"
        }
    }

},
"auth_required": true,
"api": "1.0",
"fuel-library_sha": "49c7ddeb5e4257bb52862bc5aa22600df71bb52a",
"nailgun_sha": "60f9bf536e30efd896b7b4da1830e71adda19e30",
"feature_groups":

    [
        "mirantis"
    ],
    "openstack_version": "2014.2.2-7.0",
    "production": "docker",
    "python-fuelclient_sha": "accd6493bf034ba7c70c987ace8f1dcd960cbdf5",
    "astute_sha": "9cbb8ae5adbe6e758b24b3c1021aac1b662344e8",
    "fuel-ostf_sha": "62785c16f8399f30526d24c52bb9ca23e1585bfb",
    "release": "7.0",
    "fuelmain_sha": "28551be12a050acb9a633933ed6a8b25e2dc411c"

}

[Ilya Shakhat (shakhat)]

Reproduced the same on ISO 41 (3 ctrl + 1 comp + 1 comp/cinder) during spawning Ubuntu Cloud image. The error is in nova-compute.log:
  ProcessExecutionError: Unexpected error while running command.
  Command: sudo nova-rootwrap /etc/nova/rootwrap.conf blkid -o value -s TYPE /dev/nbd0
  Exit code: 2
  Stdout: u''
  Stderr: u''

Instance based on Cirros is created successfully, so it seems the issue relates to instances with disk.

The issue blocks data-plane performance by Shaker, thus raising priority to High.

[Sergey Kolekonov (skolekonov)]

I've done some research and the problem seems to be connected with novaclient (default options or something). I wasn't able to spawn instance successfully using CLI or Heat, but it was possible using Horizon.

[Roman Podoliaka (rpodolyaka)]

Sergey, that's very unlikely, IMO, given the exact error we are seeing. I believe, it's just a coincidence. Maybe it's the first instance run which failed and subsequent ones succeeded.

[Roman Podoliaka (rpodolyaka)]

Are you guys still seeing this issue? I tried to reproduce it on #57 without any luck.

My env is ubuntu, neutron, 1 controllers + 2 computes, default storage providers (read: no ceph). I'm using trusty cloud image and custom flavor (1 VCPU, 512 RAM, 5 GB HDD).

I successfully booted instances like 10 times in a row (both injecting custom files and not) and in all cases I was able to SSH to an instance.

[Ilya Shakhat (shakhat)]

Roman, I confirm that the issue is not presented on ISO #54.

[Andrey Sledzinskiy (asledzinskiy)]

We also stopped seeing this issue in our Swarm tests

[Roman Podoliaka (rpodolyaka)]

I don't like `Invalid` status, but this doesn't seem to be `Fixed committed` either, as we simply does not know what exactly produced the error and how it has been fixed...

[Tony Breeds (o-tony)]

Is this a duplicate of: https://bugs.launchpad.net/cloud-archive/+bug/1426324
and therefore fixed with: https://review.openstack.org/#/c/159797/