nova-network bridge existing system alias handeling failiure.

Bug #1077066 reported by David Busby
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Édouard Thuleau
Havana
Fix Released
Undecided
Unassigned

Bug Description

This: https://github.com/openstack/nova/blob/stable/essex/nova/network/linux_net.py#L1033

Also prevalent in Folsom can cause nova-network to fail with the following error:

---
Command: sudo nova-rootwrap ip addr del xxx.xxx.xxx.xxx/24 brd 172.30.255.255 scope global secondary dev br0
Exit code: 255
Stdout: ''
Stderr: 'Error: either "local" is duplicate, or "secondary" is a garbage.\n'
---

This is due to the invalid passing of params to the ip addr del

irc conversation:

---
Apsu`: oneiroi: Yeah, just verified it on a box, to be sure. Can't specify the "secondary" part in the del call. It's not part of the syntax, because it's not user-controllable. A "secondary" is an IP in the same network as one already on the interface in the same scope
[4:36pm] Apsu`: Let's see if it's different in Folsom
[4:37pm] oneiroi: Apsu`: abstracted code and output http://fpaste.org/hFe3/
[4:37pm] uvirtbot`: New bug: #1077056 in tempest "images whitebox test fails with bad SQL connection URI" [High,Confirmed] https://launchpad.net/bugs/1077056
[4:37pm] AnilV4 left the chat room. (Read error: Connection reset by peer)
[4:37pm] Apsu`: Nope. Appears to do the same. https://github.com/openstack/nova/blob/stable/folsom/nova/network/linux_net.py#L1193
[4:37pm] • Apsu` nods
[4:37pm] Apsu`: Exactly the issue. It's trying to ensure_bridge, and reorder the IPs so nova's bridge IP is first
[4:38pm] maurosr left the chat room. (Quit: WeeChat 0.3.8)
[4:38pm] Apsu`: There's a comment from vish in initialize_gateway_device about the same issue
[4:39pm] Apsu`: So that field splitting code needs to filter out anything past the scope.
[4:39pm] Apsu`: It's currently using [1:-1] to skip the first and last
[4:40pm] oneiroi: indeed, filing a bug will then look at a quick fix, and submit patch
[4:40pm] Apsu`: Cool.

---

Abstracted code showing the issue:

---
#!/usr/bin/env python

str='''
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 00:15:17:aa:46:04 brd ff:ff:ff:ff:ff:ff
    inet aa.bb.cc.dd/16 brd 172.30.255.255 scope global br0
    inet xxx.xxx.xxx.xxx/24 brd 192.168.1.255 scope global br0:1
    inet6 fe80::215:17ff:feaa:4604/64 scope link
       valid_lft forever preferred_lft forever
'''

for line in str.split('\n'):
  fields = line.split()
  if fields and fields[0] == 'inet':
    params = fields[1:-1]
    print params

---

['aa.bb.cc.dd/16', 'brd', '172.30.255.255', 'scope', 'global']
['xxx.xxx.xxx.xxx/24', 'brd', '192.168.1.255', 'scope', 'global']

Revision history for this message
Yaguang Tang (heut2008) wrote :

I am curious about how this situation happens ,for in a normal usecase ,br100 is created by nova automately and we only need to config one IP on the bridge . how can we have the two IPs that have the same netmask and in a broadcast domain?
if this is not caused by nova. this should not be bug.

Changed in nova:
status: New → Incomplete
Revision history for this message
David Busby (d-busby) wrote :

In this instance the aliases are being maintained for a soon to be retired legacy setup, nova.conf is set with: `public_interface = br0`

This is the main bridged interface on the host, the issue arrises when the defined public_interface allready has aliases, per information above the command would appear to be outputting:

ip addr del aa.bb.cc.dd/16 brd 172.30.255.255 scope global dev br0

which in itself is invalid.

pseudo code:

if scope <> global
  cmd = ip addr del $1 scope $5

I am unsure as to the handeling of labels etc, I am looking into this at this time.

Thierry Carrez (ttx)
Changed in nova:
status: Incomplete → New
Revision history for this message
Vish Ishaya (vishvananda) wrote :

looks like a pretty easy fix to make it handle scope properly:

Changed in nova:
importance: Undecided → Medium
status: New → Triaged
tags: added: folsom-backport-potential low-hanging-fruit
Changed in nova:
milestone: none → grizzly-rc1
Changed in nova:
assignee: nobody → Édouard Thuleau (ethuleau)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/23966

Changed in nova:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/23966
Committed: http://github.com/openstack/nova/commit/f5a45dde519c521921240dfda835a805f1eb05b3
Submitter: Jenkins
Branch: master

commit f5a45dde519c521921240dfda835a805f1eb05b3
Author: Édouard Thuleau <email address hidden>
Date: Fri Feb 15 12:49:37 2013 +0100

    Nova network needs to take care of existing alias

    Nova network should take care of existing aliases on bridge
    interfaces when it adds address on top of the list.

    Change-Id: Ife54c9eba64906406f7c44a66b596f3162ad16b6
    Fixes: bug #1077066

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-rc1 → 2013.1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.openstack.org/70164
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0d447f6edd406547e828c592823ac6624e56e91f
Submitter: Jenkins
Branch: master

commit 0d447f6edd406547e828c592823ac6624e56e91f
Author: Xavier Queralt <email address hidden>
Date: Thu Jan 30 17:28:31 2014 +0100

    Ignore 'dynamic' addr flag on bridge configuration

    When setting the bridge up, if that flag is present but not
    ignored, the delete command will fail and may leave the
    interface and the bridge in an inconsistent state.

    Closes-Bug: #1274611
    Related-Bug: #1077066
    Change-Id: Iff65d61d724d0de5d8173ac37f7f748aa23e5d8b

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/havana)

Related fix proposed to branch: stable/havana
Review: https://review.openstack.org/116150

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/havana)

Change abandoned by Can ZHANG (<email address hidden>) on branch: stable/havana
Review: https://review.openstack.org/116150
Reason: sorry, duplicate patch

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Alan Pevec (<email address hidden>) on branch: stable/havana
Review: https://review.openstack.org/115888
Reason: https://review.openstack.org/116150 is the correct backport

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/havana)

Reviewed: https://review.openstack.org/116150
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6cbbb615787844d6d0f084e16605d30b56079c97
Submitter: Jenkins
Branch: stable/havana

commit 6cbbb615787844d6d0f084e16605d30b56079c97
Author: Xavier Queralt <email address hidden>
Date: Thu Jan 30 17:28:31 2014 +0100

    Ignore 'dynamic' addr flag on bridge configuration

    When setting the bridge up, if that flag is present but not
    ignored, the delete command will fail and may leave the
    interface and the bridge in an inconsistent state.

    Closes-Bug: #1274611
    Related-Bug: #1077066
    Change-Id: Iff65d61d724d0de5d8173ac37f7f748aa23e5d8b
    (cherry picked from commit 0d447f6edd406547e828c592823ac6624e56e91f)

tags: added: in-stable-havana
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.