OpenStack Compute (nova)

Cannot boot VM with Contrail SDN controller

Series queens
Bug #1742963

Bug #1742963 reported by Édouard Thuleau on 2018-01-12

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	High	Édouard Thuleau
	Queens	Fix Committed	High	Matt Riedemann

Bug Description

On the master branch, nova-compute fails to create vif on the Contrail vrouter compute agent and the instance fails to spawn:
...
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] six.reraise(self.type_, self.value, self.tb)
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] File "/opt/stack/openstack/nova/nova/virt/libvirt/driver.py", line 5238,
in _create_domain_and_network
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] self.plug_vifs(instance, network_info)
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] File "/opt/stack/openstack/nova/nova/virt/libvirt/driver.py", line 755, in plug_vifs
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] self.vif_driver.plug(instance, vif)
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] File "/opt/stack/openstack/nova/nova/virt/libvirt/vif.py", line 769, in p
lug
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] func(instance, vif)
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] File "/opt/stack/openstack/nova/nova/virt/libvirt/vif.py", line 727, in plug_vrouter
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] instance, vif, ip_addr, ip6_addr, ptype)
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/priv_context.py", line 207, in _wrap
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] return self.channel.remote_call(name, args, kwargs)
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py", line 192, in remote_call
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] result = self.send_recv((Message.CALL.value, name, args, kwargs))
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/comm.py", line 163, in send_recv
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] self.writer.send((myid, msg))
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/comm.py", line
54, in send
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] buf = msgpack.packb(msg, use_bin_type=True)
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] File "/usr/local/lib/python2.7/dist-packages/msgpack/__init__.py", line 47, in packb
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] return Packer(**kwargs).pack(o)
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] File "msgpack/_packer.pyx", line 231, in msgpack._packer.Packer.pack (msg
pack/_packer.cpp:3661)
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] File "msgpack/_packer.pyx", line 233, in msgpack._packer.Packer.pack (msg
pack/_packer.cpp:3503)
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] File "msgpack/_packer.pyx", line 221, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:3230)
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] File "msgpack/_packer.pyx", line 221, in msgpack._packer.Packer._pack (ms
gpack/_packer.cpp:3230)
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] File "msgpack/_packer.pyx", line 221, in msgpack._packer.Packer._pack (ms
gpack/_packer.cpp:3230)
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] File "msgpack/_packer.pyx", line 228, in msgpack._packer.Packer._pack (ms
gpack/_packer.cpp:3382)
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c] TypeError: can't serialize Instance(access_ip_v4=None,access_ip_v6=None,arc
hitecture=None,auto_disk_config=False,availability_zone='nova',cell_name=None,cleaned=False,config_drive='',created_at=2018-01-12T15:05:54Z,default_ephemeral_device=None,default_swap_device=None,deleted=False,deleted_at=None,device_metadata=None,disable_terminate=False,display_description='vm1',display_name='vm1',ec2_ids=EC2Ids,ephemeral_gb=0,ephemeral_key_uuid=None,fault=<?>,flavor=Flavor(1),host='ethuleau-contrail-mast
er',hostname='vm1',id=27,image_ref='a9c4ac60-dddd-4445-8199-02e0e88b1431',info_cache=InstanceInfoCache,instance_type_id=1,kernel_id='',key_data=None,key_name=None,keypairs=KeyPairList,launch_index=0,launched_at=None,launched_on='ethuleau-contrail-master',locked=False,locked_by=None,memory_mb=256,metadata={},migration_context=<?>,new_flavor=None,node='ethuleau-contrail-master',numa_topology=None,old_flavor=None,os_type=None,pci_devices=PciDeviceList,pci_requests=InstancePCIRequests,power_state=0,progress=0,project_id='80f750bf6b804de79bbd50ae8d31df32',ramdisk_id='',reservation_id='r-p3ur515e',root_device_name='/dev/vda',root_gb=0
,security_groups=SecurityGroupList,services=<?>,shutdown_terminate=False,system_metadata={boot_roles='anotherrole,Member',image_base_image_ref='a9c4ac60-dddd-4445-8199-02e0e88b1431',image_container_format='bare',
image_disk_format='qcow2',image_min_disk='0',image_min_ram='0',network_allocated='True',owner_project_name='demo',owner_user_name='demo'},tags=TagList,task_state='spawning',terminated_at=None,updated_at=2018-01-12T15:05:56Z,user_data=None,user_id='837fc18d88384a62b84635f358a75eee',uuid=8748627c-e31f-4c90-83e2-16abdf9c1e2c,vcpu_model=VirtCPUModel,vcpus=1,vm_mode=None,vm_state='building')
Jan 12 15:05:59 ethuleau-contrail-master nova-compute[5512]: ERROR nova.compute.manager [instance: 8748627c-e31f-4c90-83e2-16abdf9c1e2c]

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-01-12: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/533212

Changed in nova:
assignee:	nobody → Édouard Thuleau (ethuleau)
status:	New → In Progress

Matt Riedemann (mriedem) on 2018-02-21

tags:

added: queens-rc-potential

melanie witt (melwitt) on 2018-02-21

Changed in nova:
importance:	Undecided → High

Revision history for this message

melanie witt (melwitt) wrote on 2018-02-21:

This was a regression in Queens due to privsep.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-02-21: Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/546778

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-02-21: Fix merged to nova (master)

Reviewed: https://review.openstack.org/533212
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1f5fe3190bf2e0987945a6ef9ec430673c9fa736
Submitter: Zuul
Branch: master

commit 1f5fe3190bf2e0987945a6ef9ec430673c9fa736
Author: Édouard Thuleau <email address hidden>
Date: Fri Jan 12 16:20:32 2018 +0100

Update plugs Contrail methods to work with privsep

    As privsep uses the msgpack to send method arguments to the privsep
    daemon, we could not use anymore custom data type like
    nova.objects.instance.Instance.

Change-Id: I09f04d5b2f1cb39339ad7c4569186db5d361797a
Closes-Bug: #1742963

Changed in nova:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-02-22: Fix merged to nova (stable/queens)

Reviewed: https://review.openstack.org/546778
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5a646d82bad6a71da28296e3ab06dc5ce2c0f716
Submitter: Zuul
Branch: stable/queens

commit 5a646d82bad6a71da28296e3ab06dc5ce2c0f716
Author: Édouard Thuleau <email address hidden>
Date: Fri Jan 12 16:20:32 2018 +0100

Update plugs Contrail methods to work with privsep

    As privsep uses the msgpack to send method arguments to the privsep
    daemon, we could not use anymore custom data type like
    nova.objects.instance.Instance.

    Change-Id: I09f04d5b2f1cb39339ad7c4569186db5d361797a
    Closes-Bug: #1742963
    (cherry picked from commit 1f5fe3190bf2e0987945a6ef9ec430673c9fa736)