Cannot start nova-network on juno and kilo

Bug #1376596 reported by Bruno Bompastor on 2014-10-02
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Undecided
Brent Eagles

Bug Description

Hi!

I was testing packstack --allinone on "RDO test day Juno milestone 3" and i cannot start nova-network.

OS: CentOS7
Openstack version: juno-1
Packstack cmd: packstack --allinone --os-neutron-install=n --os-heat-cfn-install=y --os-heat-install=y --use-epel=y

Error:

2014-10-01 18:17:53.853 6108 AUDIT nova.service [-] Starting network node (version 2014.2-0.4.b3.el7.centos)
2014-10-01 18:17:54.055 6108 ERROR nova.openstack.common.threadgroup [-] Failed to add interface: can't add lo to bridge br100: Invalid argument
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/threadgroup.py", line 125, in wait
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup x.wait()
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/threadgroup.py", line 47, in wait
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 173, in wait
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 293, in switch
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 212, in main
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/service.py", line 492, in run_service
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup service.start()
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/service.py", line 164, in start
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup self.manager.init_host()
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/manager.py", line 1776, in init_host
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup super(FlatDHCPManager, self).init_host()
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/manager.py", line 334, in init_host
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup self._setup_network_on_host(ctxt, network)
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/manager.py", line 1785, in _setup_network_on_host
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup self._initialize_network(network)
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/manager.py", line 1451, in _initialize_network
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup self.l3driver.initialize_gateway(network)
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/l3.py", line 105, in initialize_gateway
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup gateway=(network_ref['gateway'] is not None))
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/linux_net.py", line 1411, in plug
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup return _get_interface_driver().plug(network, mac_address, gateway)
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/linux_net.py", line 1460, in plug
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup network, gateway)
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 325, in inner
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup return f(*args, **kwargs)
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/network/linux_net.py", line 1575, in ensure_bridge
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup raise exception.NovaException(msg)
2014-10-01 18:17:54.055 6108 TRACE nova.openstack.common.threadgroup NovaException: Failed to add interface: can't add lo to bridge br100: Invalid argument

Hi again!

So I found the problem.

So on juno you changed the logic on this file /usr/lib/python2.7/site-packages/nova/network/linux_net.py

RDO packages:
icehouse: python-nova.noarch 2014.1.2-1.el6 @openstack-icehouse
juno: python-nova.noarch 2014.2-0.4.b3.el7.centos @openstack-juno

icehouse ensure_bridge function:

        if interface:
            msg = _('Adding interface %(interface)s to bridge %(bridge)s')
            LOG.debug(msg, {'interface': interface, 'bridge': bridge})
            out, err = _execute('brctl', 'addif', bridge, interface,
                                check_exit_code=False, run_as_root=True)
            out, err = _execute('ip', 'link', 'set', interface, 'up',
                                check_exit_code=False, run_as_root=True)

master/juno ensure_bridge function:

        if interface:
            msg = _('Adding interface %(interface)s to bridge %(bridge)s')
            LOG.debug(msg, {'interface': interface, 'bridge': bridge})
            out, err = _execute('brctl', 'addif', bridge, interface,
                                check_exit_code=False, run_as_root=True)
            if (err and err != "device %s is already a member of a bridge; "
                     "can't enslave it to bridge %s.\n" % (interface, bridge)):
                msg = _('Failed to add interface: %s') % err
                raise exception.NovaException(msg)

            out, err = _execute('ip', 'link', 'set', interface, 'up',
                                check_exit_code=False, run_as_root=True)

I understand that the developer was trying not to lose the error message here but it didnt catch it properly.

I hope that helps!

Sean Dague (sdague) wrote :

This seems right, interface shouldn't have been 'lo' I don't think.

Changed in nova:
status: New → Invalid

Yeah you are probably right. But if i comment flat_interface on nova.conf, it will put 'lo' by default. It's weird.

Can you figure this out?

People are implementing workarounds (http://sethanil.blogspot.ch/2014/11/openstack-nova-compute-fails-to-start.html) because this is not fixed.

Setting flat_interface on nova.conf to None should work, so i think is a bug.

Changed in nova:
status: Invalid → Opinion
Changed in nova:
status: Opinion → Confirmed
summary: - Cannot start nova-network on juno-1
+ Cannot start nova-network on juno and kilo
Boris Derzhavets (bderzhavets) wrote :

I believe Nova-network was deprecated in RDO Havana . So, RDO IceHouse,Juno,Kilo don't support nova-network as well.

Tim Bell (tim-bell) wrote :

From an OpenStack perspective, Nova network has not been deprecated. See http://docs.openstack.org/openstack-ops/content/nova-network-deprecation.html. It is still used by over 30% of the current deployments.

Can you point me to where this was announced for RDO ?

Alan Pevec (apevec) wrote :

nova-network was not announced for RDO because RDO is vanilla upstream packaging and as long as upstream has it, RDO will include.
But RDO CI jobs only cover neutron networking[1] and what's not tested is broken by definition.

[1] https://prod-rdojenkins.rhcloud.com/search/?q=khaleesi-rdo-kilo-production

Alan Pevec (apevec) wrote :

Correction: nova-network _deprecation_ was not announced...

Clearly was not deprecated since neutron does not provide the same functionality (FlatDHCP for example) and there is not a migration path yet...

Concerning the bug, I think we could improve the situation in two ways:

1) Not setting the flat_interface on nova.conf should not bridge the interfaces (right now forces 'lo');
2) if NOT, check if the interface is 'lo' like here http://sethanil.blogspot.ch/2014/11/openstack-nova-compute-fails-to-start.html and not bridge the interfaces.

The origin of this bug is due to the impossibility to add loopback to the bridge on rhel 7 (reference: https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1153079)

Brent Eagles (beagles) on 2015-07-21
Changed in nova:
assignee: nobody → Brent Eagles (beagles)
Brent Eagles (beagles) wrote :

It seems packstack that is setting the flat_interface configuration to 'lo' - which is invalid. However, when I take an answerfile approach and unset the *PRIVIF configuration variable (which you can -probably SHOULD- do for an allinone setup since there is no external connectivity required for the tenant network bridge), it balks at the empty value. In any case, this is more of a packstack issue - not a nova one.

Changed in nova:
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.