Instance creation failed on Ubuntu, Neutron cluster with Unexpected error while running command.\nCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf blkid -o value -s TYPE /dev/nbd14

Bug #1473975 reported by Andrey Sledzinskiy
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Invalid
High
MOS Nova

Bug Description

Description - instance creation failed with Unexpected error while running command.\nCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf blkid -o value -s TYPE /dev/nbd14

Steps:
1. Create next cluster - Ubuntu, HA, Neutron Gre, 3 controller, 2 compute, 1 cinder
2. Deploy cluster
3. Create keypair for instance
. openrc; nova keypair-add instancekey > /root/.ssh/webserver_rsa
4. Try to create instance with keypair

Expected - instance was created
Actual - instance failed to be created

From node-1 nova-conductor.log:

2015-07-13 07:07:07 ERR nova.scheduler.utils [req-b302c9f9-5ab2-4447-9948-e1906e2c7144 3ee23604f8ce4cceb1b7f19f5d8e402c 80a4fa5ae88442c5af8b5525cd2b580f - - -] [instance: e96a177f-f54b-4f26-bac7-042234d79528] Error from last host: node-3.test.domain.local (node node-3.test.domain.local): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2219, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2362, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance e96a177f-f54b-4f26-bac7-042234d79528 was re-scheduled: Unexpected error while running command.\nCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf blkid -o value -s TYPE /dev/nbd14\nExit code: 2\nStdout: u''\nStderr: u''\n"]

After second try to create instance it was created without errors

{

    "build_id": "2015-07-12_15-52-44",
    "build_number": "31",
    "release_versions":

{

    "2014.2.2-7.0":

{

    "VERSION":

{

    "build_id": "2015-07-12_15-52-44",
    "build_number": "31",
    "api": "1.0",
    "fuel-library_sha": "49c7ddeb5e4257bb52862bc5aa22600df71bb52a",
    "nailgun_sha": "60f9bf536e30efd896b7b4da1830e71adda19e30",
    "feature_groups":

            [
                "mirantis"
            ],
            "openstack_version": "2014.2.2-7.0",
            "production": "docker",
            "python-fuelclient_sha": "accd6493bf034ba7c70c987ace8f1dcd960cbdf5",
            "astute_sha": "9cbb8ae5adbe6e758b24b3c1021aac1b662344e8",
            "fuel-ostf_sha": "62785c16f8399f30526d24c52bb9ca23e1585bfb",
            "release": "7.0",
            "fuelmain_sha": "28551be12a050acb9a633933ed6a8b25e2dc411c"
        }
    }

},
"auth_required": true,
"api": "1.0",
"fuel-library_sha": "49c7ddeb5e4257bb52862bc5aa22600df71bb52a",
"nailgun_sha": "60f9bf536e30efd896b7b4da1830e71adda19e30",
"feature_groups":

    [
        "mirantis"
    ],
    "openstack_version": "2014.2.2-7.0",
    "production": "docker",
    "python-fuelclient_sha": "accd6493bf034ba7c70c987ace8f1dcd960cbdf5",
    "astute_sha": "9cbb8ae5adbe6e758b24b3c1021aac1b662344e8",
    "fuel-ostf_sha": "62785c16f8399f30526d24c52bb9ca23e1585bfb",
    "release": "7.0",
    "fuelmain_sha": "28551be12a050acb9a633933ed6a8b25e2dc411c"

}

Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :
Changed in mos:
status: New → Confirmed
Revision history for this message
Ilya Shakhat (shakhat) wrote :

Reproduced the same on ISO 41 (3 ctrl + 1 comp + 1 comp/cinder) during spawning Ubuntu Cloud image. The error is in nova-compute.log:
  ProcessExecutionError: Unexpected error while running command.
  Command: sudo nova-rootwrap /etc/nova/rootwrap.conf blkid -o value -s TYPE /dev/nbd0
  Exit code: 2
  Stdout: u''
  Stderr: u''

Instance based on Cirros is created successfully, so it seems the issue relates to instances with disk.

The issue blocks data-plane performance by Shaker, thus raising priority to High.

Changed in mos:
importance: Medium → High
Revision history for this message
Sergey Kolekonov (skolekonov) wrote :

I've done some research and the problem seems to be connected with novaclient (default options or something). I wasn't able to spawn instance successfully using CLI or Heat, but it was possible using Horizon.

Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

Sergey, that's very unlikely, IMO, given the exact error we are seeing. I believe, it's just a coincidence. Maybe it's the first instance run which failed and subsequent ones succeeded.

Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

Are you guys still seeing this issue? I tried to reproduce it on #57 without any luck.

My env is ubuntu, neutron, 1 controllers + 2 computes, default storage providers (read: no ceph). I'm using trusty cloud image and custom flavor (1 VCPU, 512 RAM, 5 GB HDD).

I successfully booted instances like 10 times in a row (both injecting custom files and not) and in all cases I was able to SSH to an instance.

Revision history for this message
Ilya Shakhat (shakhat) wrote :

Roman, I confirm that the issue is not presented on ISO #54.

Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :

We also stopped seeing this issue in our Swarm tests

Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

I don't like `Invalid` status, but this doesn't seem to be `Fixed committed` either, as we simply does not know what exactly produced the error and how it has been fixed...

Changed in mos:
status: Confirmed → Invalid
Revision history for this message
Tony Breeds (o-tony) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.