oslo.rootwrap 1.5.0 deprecation warning brakes nova

[Vasyl Saienko]

<182>Mar 4 15:54:07 node-3 nova-compute Compute_service record updated for ('node-3.test.domain.local', 'node-3.test.domain.local')
<182>Mar 4 15:54:07 node-3 nova-compute Compute_service record updated for node-3.test.domain.local:node-3.test.domain.local
<179>Mar 4 15:54:07 node-3 nova-network Exception during message handling: Failed to add interface: /usr/bin/nova-rootwrap:6: DeprecationWarning: The oslo namespace package is deprecated. Please use oslo_rootwrap instead.
  from oslo.rootwrap.cmd import main
device eth0.103 is already a member of a bridge; can't enslave it to bridge br100.
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 137, in _dispatch_and_reply
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 180, in _dispatch
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 126, in _do_dispatch
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/floating_ips.py", line 114, in allocate_for_instance
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher **kwargs)
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 511, in allocate_for_instance
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher requested_networks=requested_networks)
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 192, in _allocate_fixed_ips
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher vpn=vpn, address=address)
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 967, in allocate_fixed_ip
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher exc_info=True)
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 943, in allocate_fixed_ip
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher self._setup_network_on_host(context, network)
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1816, in _setup_network_on_host
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher self._initialize_network(network)
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1480, in _initialize_network
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher self.l3driver.initialize_gateway(network)
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/l3.py", line 105, in initialize_gateway
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher gateway=(network_ref['gateway'] is not None))
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1403, in plug
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher return _get_interface_driver().plug(network, mac_address, gateway)
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1452, in plug
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher network, gateway)
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 272, in inner
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1567, in ensure_bridge
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher raise exception.NovaException(msg)
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher NovaException: Failed to add interface: /usr/bin/nova-rootwrap:6: DeprecationWarning: The oslo namespace package is deprecated. Please use oslo_rootwrap instead.
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher from oslo.rootwrap.cmd import main
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher device eth0.103 is already a member of a bridge; can't enslave it to bridge br100.
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher
2015-03-04 15:54:07.848 20962 TRACE oslo.messaging.rpc.dispatcher
<179>Mar 4 15:54:07 node-3 nova-network Returning exception Failed to add interface: /usr/bin/nova-rootwrap:6: DeprecationWarning: The oslo namespace package is deprecated. Please use oslo_rootwrap instead.
  from oslo.rootwrap.cmd import main
device eth0.103 is already a member of a bridge; can't enslave it to bridge br100.
 to caller

[Davanum Srinivas (DIMS) (dims-v)]

DeprecationWarning is just a warning, does not cause the cause the process to fail. It is fixed in
https://review.openstack.org/#/c/151662/

your real problem is the "device eth0.103 is already a member of a bridge" i think.

[Vasyl Saienko (vsaienko)]

fixed at: https://review.fuel-infra.org/#/c/4387/

[Vasyl Saienko (vsaienko)]

also related: https://review.fuel-infra.org/#/c/4259/

[Dmitry Mescheryakov (dmitrymex)]

The issue seems to be fixed - it didn't surface for 5 days

[Oleksiy Butenko (obutenko)]

verified on 6.0.1 build
{"build_id": "2015-03-09_09-13-55", "ostf_sha": "b38332e6741fc4b0ef40a9b4fe9d2804ee6437cc", "build_number": "125", "auth_required": true, "api": "1.0", "nailgun_sha": "83f8c83b5e77973bd391dde3b97d22999ff02bd2", "production": "docker", "fuelmain_sha": "bc1a1279509a87de0b9201e6dd2d393e0e122905", "astute_sha": "f7cda2171b0b677dfaeb59693d980a2d3ee4c3e0", "feature_groups": ["mirantis"], "release": "6.0.1", "release_versions": {"2014.2.2-6.0.1": {"VERSION": {"build_id": "2015-03-09_09-13-55", "ostf_sha": "b38332e6741fc4b0ef40a9b4fe9d2804ee6437cc", "build_number": "125", "api": "1.0", "nailgun_sha": "83f8c83b5e77973bd391dde3b97d22999ff02bd2", "production": "docker", "fuelmain_sha": "bc1a1279509a87de0b9201e6dd2d393e0e122905", "astute_sha": "f7cda2171b0b677dfaeb59693d980a2d3ee4c3e0", "feature_groups": ["mirantis"], "release": "6.0.1", "fuellib_sha": "d771c4fe4702dbf373140a4f79046e4eca7cd1a8"}}}, "fuellib_sha": "d771c4fe4702dbf373140a4f79046e4eca7cd1a8"}

[Viktor Serhieiev (vsergeyev)]

see fix in 6.1 - https://review.fuel-infra.org/#/c/4503/

[Vasyl Saienko (vsaienko)]

faced the same issue on 6.1 Centos, Neutron, 1 Controller 1 Compute

2015-03-16T09:54:24.027388+00:00 debug: Dynamic looping call <bound method Service.periodic_tasks of <nova.service.Service object at 0x3ad6150>> sleeping for 60.00 seconds
2015-03-16T09:55:24.029801+00:00 debug: Dynamic looping call <bound method Service.periodic_tasks of <nova.service.Service object at 0x3ad6150>> sleeping for 60.00 seconds
2015-03-16T09:55:39.135296+00:00 err: Error from last host: node-3.test.domain.local (node node-3.test.domain.local): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2033, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2164, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 1b2f250b-b62c-45b1-a3c4-b3413a58b0e6 was re-scheduled: Failed to add interface: /usr/bin/nova-rootwrap:6: DeprecationWarning: The oslo namespace package is deprecated. Please use oslo_rootwrap instead.\n from oslo.rootwrap.cmd import main\ndevice vlan103 is already a member of a bridge; can\'t enslave it to bridge br103.\n\nTraceback (most recent call last):\n\n File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 137, in _dispatch_and_reply\n incoming.message))\n\n File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 180, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n\n File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 126, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n\n File "/usr/lib/python2.6/site-packages/nova/network/floating_ips.py", line 114, in allocate_for_instance\n **kwargs)\n\n File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 511, in allocate_for_instance\n requested_networks=requested_networks)\n\n File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 192, in _allocate_fixed_ips\n vpn=vpn, address=address)\n\n File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1954, in allocate_fixed_ip\n self._setup_network_on_host(context, network)\n\n File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 272, in inner\n return f(*args, **kwargs)\n\n File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 2073, in _setup_network_on_host\n self._initialize_network(network)\n\n File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1480, in _initialize_network\n self.l3driver.initialize_gateway(network)\n\n File "/usr/lib/python2.6/site-packages/nova/network/l3.py", line 105, in initialize_gateway\n gateway=(network_ref[\'gateway\'] is not None))\n\n File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1403, in plug\n return _get_interface_driver().plug(network, mac_address, gateway)\n\n File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1445, in plug\n network.get(\'mtu\'))\n\n File "/usr/lib/python2.6...

[Dmitry Mescheryakov (dmitrymex)]

The issue appeared only after we updated oslo.rootwrap, it is not present in 6.0

[Oleksiy Butenko (obutenko)]

on verification

[Oleksiy Butenko (obutenko)]

verified on MOS 6.1 ISO 342
{"build_id": "2015-05-18_03-43-53", "build_number": "432", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-05-18_03-43-53", "build_number": "432", "api": "1.0", "fuel-library_sha": "1621cb350af744f497c35f2b3bb889c2041465d8", "nailgun_sha": "076566b5df37f681c3fd5b139c966d680d81e0a5", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "38765563e1a7f14f45201fd47cf507393ff5d673", "astute_sha": "cb655a9a9ad26848bcd9d9ace91857b6f4a0ec15", "fuel-ostf_sha": "9ce1800749081780b8b2a4a7eab6586583ffaf33", "release": "6.1", "fuelmain_sha": "0e970647a83d9a7d336c4cc253606d4dd0d59a60"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "1621cb350af744f497c35f2b3bb889c2041465d8", "nailgun_sha": "076566b5df37f681c3fd5b139c966d680d81e0a5", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "38765563e1a7f14f45201fd47cf507393ff5d673", "astute_sha": "cb655a9a9ad26848bcd9d9ace91857b6f4a0ec15", "fuel-ostf_sha": "9ce1800749081780b8b2a4a7eab6586583ffaf33", "release": "6.1", "fuelmain_sha": "0e970647a83d9a7d336c4cc253606d4dd0d59a60"}