amd64 bionic queens - Profile is currently in use

[Andrew McLeod]

MOSCI Deployment URL: http://mosci:8080/job/0.%20Full%20Cloud%20-%20Pipeline/4962

Bundle URL: https://raw.githubusercontent.com/openstack-charmers/openstack-bundles/master/development/openstack-lxd-bionic-queens/bundle.yaml

Instance launch error:

u'message': u'Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 4dce002e-1a15-4390-a516-85868a31dabb. Last exception: Profile is currently in use', u'code': 500, u'details': u' File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 567, in build_instances\n filter_properties, instances[0].uuid)\n File "/usr/lib/python2.7/dist-packages/nova/scheduler/utils.py", line 551, in populate_retry\n raise exception.MaxRetriesExceeded(reason=msg)\n', u'created': u'2019-03-27T09:06:39Z'}

Full crashdump:

[Sahid Orentino (sahid-ferdjaoui)]

The "Profile is currently in use" is basically an other issue that happens when the process to spawn an instance fails. We try to do some sort of cleanups [0] which are failing with that error.

The real issue which makes the process to fail looks to be related to LXD:

  Failed to run: /usr/lib/lxd/lxd forkstart instance-00000006 /var/lib/lxd/containers /var/log/lxd/instance-00000006/lxc.conf

[0] https://github.com/openstack/nova-lxd/blob/stable/queens/nova/virt/lxd/driver.py#L595

[Alex Kavanagh (ajkavanagh)]

Good catch sahid; I suspect we really ought to just catch that exception in nova-lxd, log it, and then continue, as there's not much else we can do at that point.

[Sahid Orentino (sahid-ferdjaoui)]

root@node-jaeger:~# cat /var/log/lxd/instance-00000007/lxc.log
lxc instance-00000007 20190401090620.857 WARN conf - conf.c:lxc_setup_devpts:1616 - Invalid argument - Failed to unmount old devpts instance
lxc instance-00000007 20190401090620.894 ERROR start - start.c:start:2028 - No such file or directory - Failed to exec "/sbin/init"
lxc instance-00000007 20190401090620.894 ERROR sync - sync.c:__sync_wait:62 - An error occurred in another process (expected sequence number 7)
lxc instance-00000007 20190401090620.894 WARN network - network.c:lxc_delete_network_priv:2561 - Failed to rename interface with index 47 from "eth0" to its initial name "tin030d1e93-4d"
lxc instance-00000007 20190401090620.894 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:842 - Received container state "ABORTING" instead of "RUNNING"
lxc instance-00000007 20190401090620.895 ERROR start - start.c:__lxc_start:1939 - Failed to spawn container "instance-00000007"
lxc 20190401090620.905 WARN commands - commands.c:lxc_cmd_rsp_recv:132 - Connection reset by peer - Failed to receive response for command "get_state"

I made some search based on the error reported by LXD during the spawning process. It seems that the image used is an ubuntu-amd64 not compatible with LXD. I just uploaded one xenial referenced in the README of the project [0] and all seem to work.

I'm still going to push a patch at least to fix the way we hide the root issue.

[0] https://github.com/openstack/nova-lxd

[Sahid Orentino (sahid-ferdjaoui)]

commit 067e4bd1bf3a84f2d2558b8863628371bdfb8e18 (HEAD -> master, origin/master, origin/HEAD)
Author: Sahid Orentino Ferdjaoui <email address hidden>
Date: Tue Apr 2 11:06:34 2019 +0200

    driver: catch any cleanup exceptions during spawning process

    If something wrong happens during the process of spawning an
    instance. The process is calling 'cleanup' to rollback, unfortunately
    this process is also sensible and can fail in a way that the original
    error would be hidden.

    In this commit we ensure to catch any exceptions coming from cleanup
    to keep the original and root error well logged.

    Change-Id: I81e5d0a9532e88b11ad2ef14c995db6e6a966f2f
    Related-bug: #1821878
    Signed-off-by: Sahid Orentino Ferdjaoui <email address hidden>