OpenStack Compute (nova)

LibvirtBridgeDriver crashes when spawning an instance with NoopFirewallDriver

Bug #1050433 reported by Mathieu Rohon
28
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Daniel Berrange
OpenStack Compute (nova) 2013.1 "grizzly"
Folsom
Fix Released
High
Gary Kotton
OpenStack Compute (nova) 2012.2.4

Bug Description

I am trying the LibvirtHybridOVSBridgeDriver, with the uptodate devsatck and the vlan manager.

I've created one network for the demo project :
nova-manage network create net2 --fixed_range_v4=172.16.2.0/24 --num_networks=1 --network_size=256 --vlan=1001

I boot an instance :
nova boot --flavor 1 --image 000f2e5b-e1d1-4bc0-a9d1-3e07d527f7f1 vm_1

the instance can't be launched du to the following error :

2012-09-13 15:14:06 ERROR nova.compute.manager [req-a10f2d0d-4992-4b50-a297-b7b188ab98a1 demo demo] [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] Instance failed to spawn
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] Traceback (most recent call last):
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] File "/opt/stack/nova/nova/compute/manager.py", line 735, in _spawn
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] block_device_info)
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] File "/opt/stack/nova/nova/exception.py", line 117, in wrapped
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] temp_level, payload)
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] self.gen.next()
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] File "/opt/stack/nova/nova/exception.py", line 92, in wrapped
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] return f(*args, **kw)
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1049, in spawn
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] block_device_info)
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1878, in _create_domain_and_network
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] self.firewall_driver.setup_basic_filtering(instance, network_info)
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] AttributeError: 'LibvirtHybridOVSBridgeDriver' object has no attribute 'setup_basic_filtering'
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f]
Instance failed to spawn
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] Traceback (most recent call last):
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] File "/opt/stack/nova/nova/compute/manager.py", line 735, in _spawn
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] block_device_info)
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] File "/opt/stack/nova/nova/exception.py", line 117, in wrapped
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] temp_level, payload)
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] self.gen.next()
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] File "/opt/stack/nova/nova/exception.py", line 92, in wrapped
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] return f(*args, **kw)
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1049, in spawn
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] block_device_info)
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1878, in _create_domain_and_network
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] self.firewall_driver.setup_basic_filtering(instance, network_info)
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f] AttributeError: 'LibvirtHybridOVSBridgeDriver' object has no attribute 'setup_basic_filtering'
2012-09-13 15:14:06 TRACE nova.compute.manager [instance: b6fab81d-d700-4a4a-bb3f-88422e3bc40f]

Revision history for this message
Mathieu Rohon (mathieu-rohon) wrote :
Download full text (4.2 KiB)

I've changed My localrc with :

LIBVIRT_FIREWALL_DRIVER=nova.virt.firewall.NoopFirewallDriver
NOVA_VIF_DRIVER=nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver

But I still have an error :

libvir: Network Filter error : Network filter not found: Could not find filter 'nova-instance-instance-00000001-fa163e40ec0f'
2012-09-13 17:52:23 ERROR nova.compute.manager [req-a45ccf42-1341-4c8f-8d47-25f22a2d446e demo demo] [instance: 3fd27ec1-b6a8-4210-b78e-6ecdb7828699] Instance failed to spawn
2012-09-13 17:52:23 TRACE nova.compute.manager [instance: 3fd27ec1-b6a8-4210-b78e-6ecdb7828699] Traceback (most recent call last):
2012-09-13 17:52:23 TRACE nova.compute.manager [instance: 3fd27ec1-b6a8-4210-b78e-6ecdb7828699] File "/opt/stack/nova/nova/compute/manager.py", line 740, in _spawn
2012-09-13 17:52:23 TRACE nova.compute.manager [instance: 3fd27ec1-b6a8-4210-b78e-6ecdb7828699] block_device_info)
2012-09-13 17:52:23 TRACE nova.compute.manager [instance: 3fd27ec1-b6a8-4210-b78e-6ecdb7828699] File "/opt/stack/nova/nova/exception.py", line 117, in wrapped
2012-09-13 17:52:23 TRACE nova.compute.manager [instance: 3fd27ec1-b6a8-4210-b78e-6ecdb7828699] temp_level, payload)
2012-09-13 17:52:23 TRACE nova.compute.manager [instance: 3fd27ec1-b6a8-4210-b78e-6ecdb7828699] File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-09-13 17:52:23 TRACE nova.compute.manager [instance: 3fd27ec1-b6a8-4210-b78e-6ecdb7828699] self.gen.next()
2012-09-13 17:52:23 TRACE nova.compute.manager [instance: 3fd27ec1-b6a8-4210-b78e-6ecdb7828699] File "/opt/stack/nova/nova/exception.py", line 92, in wrapped
2012-09-13 17:52:23 TRACE nova.compute.manager [instance: 3fd27ec1-b6a8-4210-b78e-6ecdb7828699] return f(*args, **kw)
2012-09-13 17:52:23 TRACE nova.compute.manager [instance: 3fd27ec1-b6a8-4210-b78e-6ecdb7828699] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1049, in spawn
2012-09-13 17:52:23 TRACE nova.compute.manager [instance: 3fd27ec1-b6a8-4210-b78e-6ecdb7828699] block_device_info)
2012-09-13 17:52:23 TRACE nova.compute.manager [instance: 3fd27ec1-b6a8-4210-b78e-6ecdb7828699] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1880, in _create_domain_and_network
2012-09-13 17:52:23 TRACE nova.compute.manager [instance: 3fd27ec1-b6a8-4210-b78e-6ecdb7828699] domain = self._create_domain(xml)
2012-09-13 17:52:23 TRACE nova.compute.manager [instance: 3fd27ec1-b6a8-4210-b78e-6ecdb7828699] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1859, in _create_domain
2012-09-13 17:52:23 TRACE nova.compute.manager [instance: 3fd27ec1-b6a8-4210-b78e-6ecdb7828699] domain.createWithFlags(launch_flags)
2012-09-13 17:52:23 TRACE nova.compute.manager [instance: 3fd27ec1-b6a8-4210-b78e-6ecdb7828699] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit
2012-09-13 17:52:23 TRACE nova.compute.manager [instance: 3fd27ec1-b6a8-4210-b78e-6ecdb7828699] result = proxy_call(self._autowrap, f, *args, **kwargs)
2012-09-13 17:52:23 TRACE nova.compute.manager [instance: 3fd27ec1-b6a8-4210-b78e-6ecdb7828699] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_cal...

Read more...

Revision history for this message
Mathieu Rohon (mathieu-rohon) wrote :

I have exactly the same issue wit FlatDHCPManager

Revision history for this message
Mathieu Rohon (mathieu-rohon) wrote :

it seems to work better with :

LIBVIRT_FIREWALL_DRIVER=nova.virt.libvirt.firewall.IptablesFirewallDriver

Revision history for this message
dan wendlandt (danwent) wrote :

interesting. this actually seems to be an issue with the standard LibvirtBridgeDriver, which the LibvirtHybridOVSBridgeDriver inherits from. Can you run your tests with:

LIBVIRT_FIREWALL_DRIVER=nova.virt.firewall.NoopFirewallDriver
NOVA_VIF_DRIVER=nova.virt.libvirt.vif.LibvirtBridgeDriver

I suspect you would see the same thing.

Note: these vif-drivers are meant only to be used with Quantum, and in particular the Hybrid driver assumes that open vswitch is installed, and that an openvswitch bridge br-int exists.

dan wendlandt (danwent)
summary: - LibvirtHybridOVSBridgeDriver crashes when spawning an instance
+ LibvirtBridgeDriver crashes when spawning an instance with
+ NoopFirewallDriver
dan wendlandt (danwent)
Changed in nova:
status: New → Confirmed
Russell Bryant (russellb)
Changed in nova:
importance: Undecided → High
Revision history for this message
brent salisbury (brent-salisbury) wrote :
Download full text (4.7 KiB)

I have been grinding up against this one for a couple days now also. Not sure if its the same or even not a storage issue but figured I would post details.

Hosts boot, things look great but Arps seemed to get snuffed out by the action, so went to noop.firewall. Then everything goes south.

localrc:
HOST_IP=127.0.0.1
ADMIN_PASSWORD=openstack
MYSQL_PASSWORD=openstack
RABBIT_PASSWORD=openstack
SERVICE_PASSWORD=openstack
MYSQL_PASSWORD=password
#SCHEDULER=nova.scheduler.simple.SimpleScheduler
ENABLED_SERVICES+=,-n-vol,c-api,c-sch,c-vol,-n-net,q-svc,q-agt,q-dhcp,quantum,horizon,mysql
FIXED_RANGE=172.24.17.0/24
FLOATING_RANGE=172.31.246.128/25
SYSLOG=True
LIBVIRT_FIREWALL_DRIVER=nova.virt.libvirt.firewall.IptablesFirewallDriver

Trying to get around Arps being dropped from controller to host.
Controller = 172.24.17.220
VM = 172.24.17.3

From 2 differnet phy hosts w/same config:
in_port(0),eth(src=66:70:9c:83:be:42,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=172.24.17.220,tip=172.24.17.3,op=1,sha=66:70:9c:83:be:42,tha=00:00:00:00:00:00), packets:73, bytes:3066, used:0.840s, actions:drop

in_port(0),eth(src=e8:39:35:c4:70:f4,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=172.24.17.225,tip=172.24.17.130,op=1,sha=e8:39:35:c4:70:f4,tha=00:00:00:00:00:00), packets:1, bytes:42, used:0.825s, actions:2
in_port(2),eth(src=00:1e:c9:dd:bd:15,dst=01:80:c2:00:00:00), packets:95, bytes:5700, used:0.225s, actions:drop

======================================================================

Paste is with
HOST_IP=127.0.0.1
ADMIN_PASSWORD=openstack
MYSQL_PASSWORD=openstack
RABBIT_PASSWORD=openstack
SERVICE_PASSWORD=openstack
MYSQL_PASSWORD=password
#SCHEDULER=nova.scheduler.simple.SimpleScheduler
ENABLED_SERVICES+=,-n-vol,c-api,c-sch,c-vol,-n-net,q-svc,q-agt,q-dhcp,quantum,horizon,mysql
FIXED_RANGE=172.24.17.128/25
FLOATING_RANGE=x.x.139.40/29
SYSLOG=True
LIBVIRT_FIREWALL_DRIVER=nova.virt.firewall.NoopFirewallDriver

http://pastebin.com/4pGvuWma

===============================================================================
With this config. I noticed block error which may be due to simple scheduler being depreciated w/Cinder.

Oct 2 02:44:28 openstack2 kernel: [20294.298311] block nbd15: NBD_DISCONNECT
Oct 2 02:44:28 openstack2 kernel: [20294.298430] block nbd15: Receive control failed (result -32)
Oct 2 02:44:28 openstack2 kernel: [20294.298448] block nbd15: queue cleared

HOST_IP=127.0.0.1
ADMIN_PASSWORD=openstack
MYSQL_PASSWORD=openstack
RABBIT_PASSWORD=openstack
SERVICE_PASSWORD=openstack
MYSQL_PASSWORD=password
SCHEDULER=nova.scheduler.simple.SimpleScheduler
ENABLED_SERVICES+=,-n-vol,c-api,c-sch,c-vol,-n-net,q-svc,q-agt,q-dhcp,quantum,horizon,mysql
FIXED_RANGE=172.24.17.128/25
FLOATING_RANGE=x.x.139.40/29
SYSLOG=True
LIBVIRT_FIREWALL_DRIVER=nova.virt.firewall.NoopFirewallDriver
NOVA_VIF_DRIVER=nova.virt.libvirt.vif.LibvirtBridgeDriver

libvirt.log
2012-10-02 06:39:11.719+0000: 31727: error : virNetDevGetIndex:656 : Unable to get index for interface vnet0: No such device
2012-10-02 06:42:06.644+0000: 31724: error : virNetSocketReadWire:996 : End of file while reading data: Input/output error
2012-10-02 06:42:06.644+0000: 31724: error : virNetSocke...

Read more...

OpenStack Infra (hudson-openstack)
Changed in nova:
assignee: nobody → Daniel Berrange (berrange)
status: Confirmed → In Progress
Revision history for this message
dan wendlandt (danwent) wrote :

nachi says that this bug will be fixed by: https://review.openstack.org/#/c/19126/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/19126
Committed: http://github.com/openstack/nova/commit/3f513aa1e408f762867d9cd7c64ffde68f960437
Submitter: Jenkins
Branch: master

commit 3f513aa1e408f762867d9cd7c64ffde68f960437
Author: Daniel P. Berrange <email address hidden>
Date: Mon Dec 17 20:27:50 2012 +0000

    Merge LibvirtHybridOVSBridgeDriver into LibvirtGenericVIFDriver

    The LibvirtHybridOVSBridgeDriver VIF driver and the
    LibvirtOpenVswitchDriver VIF driver both support OpenVswitch
    based guest configs. The difference is that the hybrid driver
    does not connect the VM directly to the OVS bridge. Instead it
    connects the guest to a private Linux bridge, connected to the
    OVS bridge via a veth pair. This allows iptables firewall rules
    to be applied to the guest VIF. The hybrid driver should always
    be used unless the admin has disabled Nova firewalls using the
    NoopFirewallDriver impl. Rather than require the admin to figure
    this out themselves, pick the right impl automatically based
    on the configured firewall driver.

    When Quantum gains support for firewalling, it will need to be
    able to inform Nova whether it has got a filtering applied to
    the VIFs, and thus let Nova intelligently skip its own attempt
    at filtering.

    The LibvirtGenericVIFDriver class can use the 'vif_type' mapping
    field to determine whether an OVS network configuration is
    required. In combination with a check against the firewall
    driver, a dedicated driver for OVS hybrid setup is no longer
    required.

    The LibvirtHybridOVSBridgeDriver class functionality is
    merged into LibvirtGenericVIFDriver. For backwards compatibility
    with the Folsom release, the existing LibvirtHybridOVSBridgeDriver
    class is made to inherit from LibvirtGenericVIFDriver and directly
    call the bridge specific setup APIs. This eases migration to
    the new VIF impl during the Grizzly deployment lifecycle, with
    an expectation that the LibvirtHybridOVSBridgeDriver stub
    will be deleted after the Havana release.

    Bug: 1050433
    Blueprint: libvirt-vif-driver
    Change-Id: I93294409a75aaa15d297471ab3e600c3c124bcde
    Signed-off-by: Daniel P. Berrange <email address hidden>

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/folsom)

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/22136

Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-3
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/folsom)

Reviewed: https://review.openstack.org/22136
Committed: http://github.com/openstack/nova/commit/ecd98d2555e2bf606e9078fcf4bd38f95abaaa69
Submitter: Jenkins
Branch: stable/folsom

commit ecd98d2555e2bf606e9078fcf4bd38f95abaaa69
Author: Gary Kotton <email address hidden>
Date: Sun Feb 17 14:30:01 2013 +0000

    Enable libvirt to work with NoopFirewallDriver

    Fixes bug 1050433

    Change-Id: I49613c7d1e6b14411dcdc342366e163a21673f78

Vish Ishaya (vishvananda)
Changed in nova:
assignee: Daniel Berrange (berrange) → Chet Burgess (cfb-n)
status: Fix Released → Triaged
assignee: Chet Burgess (cfb-n) → Daniel Berrange (berrange)
status: Triaged → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-3 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.