Cannot start nova-network on juno and kilo

[Bruno Bompastor]

Hi!

I was testing packstack --allinone on "RDO test day Juno milestone 3" and i cannot start nova-network.

OS: CentOS7
Openstack version: juno-1
Packstack cmd: packstack --allinone --os-neutron-install=n --os-heat-cfn-install=y --os-heat-install=y --use-epel=y

Error:

2014-10-01 18:17:53.853 6108 AUDIT nova.service [-] Starting network node (version 2014.2-0.4.b3.el7.centos)
2014-10-01 18:17:54.055 6108 ERROR nova.openstack.common.threadgroup [-] Failed to add interface: can't add lo to bridge br100: Invalid argument
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/threadgroup.py", line 125, in wait
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup x.wait()
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/threadgroup.py", line 47, in wait
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 173, in wait
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 293, in switch
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 212, in main
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/service.py", line 492, in run_service
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup service.start()
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/service.py", line 164, in start
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup self.manager.init_host()
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/manager.py", line 1776, in init_host
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup super(FlatDHCPManager, self).init_host()
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/manager.py", line 334, in init_host
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup self._setup_network_on_host(ctxt, network)
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/manager.py", line 1785, in _setup_network_on_host
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup self._initialize_network(network)
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/manager.py", line 1451, in _initialize_network
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup self.l3driver.initialize_gateway(network)
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/l3.py", line 105, in initialize_gateway
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup gateway=(network_ref['gateway'] is not None))
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/linux_net.py", line 1411, in plug
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup return _get_interface_driver().plug(network, mac_address, gateway)
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/linux_net.py", line 1460, in plug
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup network, gateway)
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 325, in inner
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup return f(*args, **kwargs)
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/linux_net.py", line 1575, in ensure_bridge
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup raise exception.NovaException(msg)
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup NovaException: Failed to add interface: can't add lo to bridge br100: Invalid argument

[Bruno Bompastor (bruno-bompastor)]

Hi again!

So I found the problem.

So on juno you changed the logic on this file /usr/lib/python2.7/site-packages/nova/network/linux_net.py

RDO packages:
icehouse: python-nova.noarch 2014.1.2-1.el6 @openstack-icehouse
juno: python-nova.noarch 2014.2-0.4.b3.el7.centos @openstack-juno

icehouse ensure_bridge function:

        if interface:
            msg = _('Adding interface %(interface)s to bridge %(bridge)s')
            LOG.debug(msg, {'interface': interface, 'bridge': bridge})
            out, err = _execute('brctl', 'addif', bridge, interface,
                                check_exit_code=False, run_as_root=True)
            out, err = _execute('ip', 'link', 'set', interface, 'up',
                                check_exit_code=False, run_as_root=True)

master/juno ensure_bridge function:

        if interface:
            msg = _('Adding interface %(interface)s to bridge %(bridge)s')
            LOG.debug(msg, {'interface': interface, 'bridge': bridge})
            out, err = _execute('brctl', 'addif', bridge, interface,
                                check_exit_code=False, run_as_root=True)
            if (err and err != "device %s is already a member of a bridge; "
                     "can't enslave it to bridge %s.\n" % (interface, bridge)):
                msg = _('Failed to add interface: %s') % err
                raise exception.NovaException(msg)

            out, err = _execute('ip', 'link', 'set', interface, 'up',
                                check_exit_code=False, run_as_root=True)

I understand that the developer was trying not to lose the error message here but it didnt catch it properly.

I hope that helps!

[Sean Dague (sdague)]

This seems right, interface shouldn't have been 'lo' I don't think.

[Bruno Bompastor (bruno-bompastor)]

Yeah you are probably right. But if i comment flat_interface on nova.conf, it will put 'lo' by default. It's weird.

[Bruno Bompastor (bruno-bompastor)]

Can you figure this out?

People are implementing workarounds (http://sethanil.blogspot.ch/2014/11/openstack-nova-compute-fails-to-start.html) because this is not fixed.

Setting flat_interface on nova.conf to None should work, so i think is a bug.

[Boris Derzhavets (bderzhavets)]

I believe Nova-network was deprecated in RDO Havana . So, RDO IceHouse,Juno,Kilo don't support nova-network as well.

[Tim Bell (tim-bell)]

From an OpenStack perspective, Nova network has not been deprecated. See http://docs.openstack.org/openstack-ops/content/nova-network-deprecation.html. It is still used by over 30% of the current deployments.

Can you point me to where this was announced for RDO ?

[Alan Pevec (apevec)]

nova-network was not announced for RDO because RDO is vanilla upstream packaging and as long as upstream has it, RDO will include.
But RDO CI jobs only cover neutron networking[1] and what's not tested is broken by definition.

[1] https://prod-rdojenkins.rhcloud.com/search/?q=khaleesi-rdo-kilo-production

[Alan Pevec (apevec)]

Correction: nova-network _deprecation_ was not announced...

[Bruno Bompastor (bruno-bompastor)]

Clearly was not deprecated since neutron does not provide the same functionality (FlatDHCP for example) and there is not a migration path yet...

Concerning the bug, I think we could improve the situation in two ways:

1) Not setting the flat_interface on nova.conf should not bridge the interfaces (right now forces 'lo');
2) if NOT, check if the interface is 'lo' like here http://sethanil.blogspot.ch/2014/11/openstack-nova-compute-fails-to-start.html and not bridge the interfaces.

The origin of this bug is due to the impossibility to add loopback to the bridge on rhel 7 (reference: https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1153079)

[Brent Eagles (beagles)]

It seems packstack that is setting the flat_interface configuration to 'lo' - which is invalid. However, when I take an answerfile approach and unset the *PRIVIF configuration variable (which you can -probably SHOULD- do for an allinone setup since there is no external connectivity required for the tenant network bridge), it balks at the empty value. In any case, this is more of a packstack issue - not a nova one.