nova can't schedule instance with libvirt-xen driver

[Boris Derzhavets]

This issue is closely related to bug https://bugs.launchpad.net/nova/+bug/1425115.

Applying patch
$ git fetch https://review.openstack.org/openstack/nova refs/changes/06/159106/5 && git checkout FETCH_HEAD
in /opt/stack/nova

I was able to shutdown and restart system via ./rejoin-stack.sh with nova-compute up and running

ubuntu@ubuntu-wks:~/devstack$ nova service-list
+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+-------------------------
| 1 | nova-conductor | ubuntu-wks | internal | enabled | up | 2015-04-02T11:06:02.000000 | - |
| 2 | nova-cert | ubuntu-wks | internal | enabled | up | 2015-04-02T11:06:02.000000 | - |
| 3 | nova-scheduler | ubuntu-wks | internal | enabled | up | 2015-04-02T11:06:02.000000 | - |
| 4 | nova-consoleauth | ubuntu-wks | internal | enabled | up | 2015-04-02T11:06:02.000000 | - |
| 5 | nova-compute | ubuntu-wks | nova | enabled | up | 2015-04-02T11:06:02.000000 | - |
+----+------------------+------------+----------+---------+-------+----------------------------+----------------------------------------------

However, attempt to launch instance fails, n-sch.log reports :-

2015-04-02 06:16:51.248 ^[[00;36mINFO nova.filters [^[[00;36m-^[[00;36m] ^[[01;35m^[[00;36mFilter RetryFilter returned 0 hosts^[[00m^M

My local.conf

[[local|localrc]]
HOST_IP=192.168.1.50
ADMIN_PASSWORD=secret
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=a682f596-76f3-11e3-b3b2-e716f9080d50

FLOATING_RANGE=192.168.10.0/24
FLAT_INTERFACE=eth0
Q_FLOATING_ALLOCATION_POOL=start=192.168.10.150,end=192.168.10.254
PUBLIC_NETWORK_GATEWAY=192.168.10.15

# Useful logging options for debugging:
DEST=/opt/stack
LOGFILE=$DEST/logs/stack.sh.log
SCREEN_LOGDIR=$DEST/logs/screen

# The default fixed range (10.0.0.0/24) conflicted with an address
# range I was using locally.
FIXED_RANGE=10.254.1.0/24
NETWORK_GATEWAY=10.254.1.1

# Services
disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service horizon
disable_service tempest

# This is a Xen Project host:
LIBVIRT_TYPE=xen

[Boris Derzhavets (bderzhavets)]

/var/log/libvirt/libvirtd.log

root@ubuntu-wks:/var/log/libvirt# tail -f libvirtd.log

2015-04-02 10:02:35.124+0000: 4118: error : libxlVmStart:787 : internal error: libxenlight failed to create new domain 'instance-00000004'
2015-04-02 10:06:46.087+0000: 4117: error : libxlVmStart:787 : internal error: libxenlight failed to create new domain 'instance-00000005'
2015-04-02 10:16:50.297+0000: 4114: error : libxlVmStart:787 : internal error: libxenlight failed to create new domain 'instance-00000006'
2015-04-02 10:36:43.078+0000: 4114: error : libxlVmStart:787 : internal error: libxenlight failed to create new domain 'instance-00000007'
2015-04-02 10:53:16.654+0000: 4116: error : libxlVmStart:787 : internal error: libxenlight failed to create new domain 'instance-00000008'
2015-04-02 16:48:15.942+0000: 4115: error : libxlVmStart:787 : internal error: libxenlight failed to create new domain 'instance-00000009'
2015-04-02 17:16:15.834+0000: 4118: error : libxlVmStart:787 : internal error: libxenlight failed to create new domain 'instance-0000000a'
2015-04-02 17:17:58.929+0000: 4118: error : libxlVmStart:787 : internal error: libxenlight failed to create new domain 'instance-0000000b'

[Boris Derzhavets (bderzhavets)]

Upload /var/log/libvirt/libxl/instance-0000000b.log

[Boris Derzhavets (bderzhavets)]

Created link due to previous report:-

ln -s /usr/lib/xen-4.4/bin/pygrub /usr/bin/pygrub

/var/log/libvirt/libxl/instance-0000000d.log looks different , but contains errors again.

libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/vif-bridge online [-1] exited with error status 1
libxl: error: libxl_device.c:1085:device_hotplug_child_death_cb: script: ip link set vif10.0 name tap1c16ef17-68 failed
libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x7f4dd0793b80: deregister unregistered
libxl: error: libxl_create.c:1226:domcreate_attach_vtpms: unable to add nic devices

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Boris Derzhavets (bderzhavets)]

Reproduced with same errors via devstack with nova.git already patched per https://bugs.launchpad.net/nova/+bug/1425115. No manual intervention from my side.

[Boris Derzhavets (bderzhavets)]

Issue was escalated to Xen-devel mailing list. Libvirt-xen driver for nova has not been tested with Neutron.
Details and xen-hotplug.log here http://lists.xen.org/archives/html/xen-devel/2015-04/msg01856.html

[Matt Riedemann (mriedem)]

Is this still an issue? What version of nova was this? Are there errors in the nova-compute logs from the build failure?

[Launchpad Janitor (janitor)]

[Expired for nova (Ubuntu) because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]