amd64 bionic queens - Profile is currently in use

Bug #1821878 reported by Andrew McLeod
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
LXD Charm
Fix Released
Undecided
Sahid Orentino

Bug Description

MOSCI Deployment URL: http://mosci:8080/job/0.%20Full%20Cloud%20-%20Pipeline/4962

Bundle URL: https://raw.githubusercontent.com/openstack-charmers/openstack-bundles/master/development/openstack-lxd-bionic-queens/bundle.yaml

Instance launch error:

u'message': u'Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 4dce002e-1a15-4390-a516-85868a31dabb. Last exception: Profile is currently in use', u'code': 500, u'details': u' File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 567, in build_instances\n filter_properties, instances[0].uuid)\n File "/usr/lib/python2.7/dist-packages/nova/scheduler/utils.py", line 551, in populate_retry\n raise exception.MaxRetriesExceeded(reason=msg)\n', u'created': u'2019-03-27T09:06:39Z'}

Full crashdump:

Revision history for this message
Sahid Orentino (sahid-ferdjaoui) wrote :

The "Profile is currently in use" is basically an other issue that happens when the process to spawn an instance fails. We try to do some sort of cleanups [0] which are failing with that error.

The real issue which makes the process to fail looks to be related to LXD:

  Failed to run: /usr/lib/lxd/lxd forkstart instance-00000006 /var/lib/lxd/containers /var/log/lxd/instance-00000006/lxc.conf

[0] https://github.com/openstack/nova-lxd/blob/stable/queens/nova/virt/lxd/driver.py#L595

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

Good catch sahid; I suspect we really ought to just catch that exception in nova-lxd, log it, and then continue, as there's not much else we can do at that point.

Changed in charm-lxd:
assignee: nobody → Sahid Orentino (sahid-ferdjaoui)
Revision history for this message
Sahid Orentino (sahid-ferdjaoui) wrote :

root@node-jaeger:~# cat /var/log/lxd/instance-00000007/lxc.log
lxc instance-00000007 20190401090620.857 WARN conf - conf.c:lxc_setup_devpts:1616 - Invalid argument - Failed to unmount old devpts instance
lxc instance-00000007 20190401090620.894 ERROR start - start.c:start:2028 - No such file or directory - Failed to exec "/sbin/init"
lxc instance-00000007 20190401090620.894 ERROR sync - sync.c:__sync_wait:62 - An error occurred in another process (expected sequence number 7)
lxc instance-00000007 20190401090620.894 WARN network - network.c:lxc_delete_network_priv:2561 - Failed to rename interface with index 47 from "eth0" to its initial name "tin030d1e93-4d"
lxc instance-00000007 20190401090620.894 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:842 - Received container state "ABORTING" instead of "RUNNING"
lxc instance-00000007 20190401090620.895 ERROR start - start.c:__lxc_start:1939 - Failed to spawn container "instance-00000007"
lxc 20190401090620.905 WARN commands - commands.c:lxc_cmd_rsp_recv:132 - Connection reset by peer - Failed to receive response for command "get_state"

I made some search based on the error reported by LXD during the spawning process. It seems that the image used is an ubuntu-amd64 not compatible with LXD. I just uploaded one xenial referenced in the README of the project [0] and all seem to work.

I'm still going to push a patch at least to fix the way we hide the root issue.

[0] https://github.com/openstack/nova-lxd

Revision history for this message
Sahid Orentino (sahid-ferdjaoui) wrote :

commit 067e4bd1bf3a84f2d2558b8863628371bdfb8e18 (HEAD -> master, origin/master, origin/HEAD)
Author: Sahid Orentino Ferdjaoui <email address hidden>
Date: Tue Apr 2 11:06:34 2019 +0200

    driver: catch any cleanup exceptions during spawning process

    If something wrong happens during the process of spawning an
    instance. The process is calling 'cleanup' to rollback, unfortunately
    this process is also sensible and can fail in a way that the original
    error would be hidden.

    In this commit we ensure to catch any exceptions coming from cleanup
    to keep the original and root error well logged.

    Change-Id: I81e5d0a9532e88b11ad2ef14c995db6e6a966f2f
    Related-bug: #1821878
    Signed-off-by: Sahid Orentino Ferdjaoui <email address hidden>

Changed in charm-lxd:
status: New → Fix Committed
James Page (james-page)
Changed in charm-lxd:
milestone: none → 19.04
David Ames (thedac)
Changed in charm-lxd:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.