[nailgun] Bonding conf is inconsistent after cloud ops

Bug #1496279 reported by Michael Polenchuk
32
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Michael Polenchuk
7.0.x
Fix Released
High
Michael Polenchuk

Bug Description

Description of the environment:
env {
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  build_number: "292"
}

network with vlan {
  bond0 { eth0, eth1 } consists of { admin/pxe }
  bond1 { eth2, eth3, eth4, eth5 } consists of {public, storage, mgmt, private}
}

Steps to reproduce:
* start deploy
* emulate error on node by interrupting the process (e.g. "killall puppet")

Expected result:
network configuration is the same as initially before deployment

Actual result:
nailgun has only two interfaces of six in "node_nic_interfaces" table.
The root cauze is that all interfaces in bond node have the same mac-addresses.

Solution:
* Block network update in nailgun for Error node state.
* Handle the identical mac-address for interfaces.
* Substitute changed mac of interface with the original one in nailgun-agent:
    # macchanger -s eth5
    Current MAC: 64:e0:04:1c:14:03 (unknown)
    Permanent MAC: 64:53:0a:49:e6:95 (unknown)

Impact:
The initial network configuration is messed up and lead to inconsistent behaviour.

Tags: area-python
Changed in fuel:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Fuel Python Team (fuel-python)
milestone: none → 7.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-nailgun-agent (master)

Fix proposed to branch: master
Review: https://review.openstack.org/223991

Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Michael Polenchuk (mpolenchuk)
status: Confirmed → In Progress
Dmitry Pyzhov (dpyzhov)
no longer affects: fuel/8.0.x
Revision history for this message
Dmitry Klenov (dklenov) wrote :

We would need to port the fix to 7.0

Dmitry Klenov (dklenov)
tags: added: 70mu1-confirmed
Revision history for this message
Michael Polenchuk (mpolenchuk) wrote :

Debug process is in progress

Dmitry Pyzhov (dpyzhov)
tags: added: area-python
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-nailgun-agent (master)

Reviewed: https://review.openstack.org/223991
Committed: https://git.openstack.org/cgit/openstack/fuel-nailgun-agent/commit/?id=e623c89f94911c5bc70eb73abf77921c7a5e246c
Submitter: Jenkins
Branch: master

commit e623c89f94911c5bc70eb73abf77921c7a5e246c
Author: Michael Polenchuk <email address hidden>
Date: Wed Sep 16 12:40:15 2015 +0300

    Return original interface mac

    In order to prevent nailgun get inconsistent network configuration in
    case of interface bonding mode return original mac address.

    Change-Id: Ib0a5c86484d86bf07098a51de87baae931db66ec
    Closes-Bug: #1496279

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Michael Polenchuk (mpolenchuk) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-nailgun-agent (stable/7.0)

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/238883

tags: added: on-verification
Revision history for this message
Ksenia Svechnikova (kdemina) wrote :

Verify the fix at MOS 8.0 https://paste.mirantis.net/show/1331/

Steps to reproduce:
* Start deploy
* Emulate error on node by interrupting the process (e.g. "killall puppet")
* Check the bonds on the nodes after it get error state with /proc/net/bond*
* Reset env, check the interface configuraion of the env

Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-nailgun-agent (stable/7.0)

Reviewed: https://review.openstack.org/238883
Committed: https://git.openstack.org/cgit/openstack/fuel-nailgun-agent/commit/?id=d6756e467dc79d08e0be75d41951bed943ac8887
Submitter: Jenkins
Branch: stable/7.0

commit d6756e467dc79d08e0be75d41951bed943ac8887
Author: Michael Polenchuk <email address hidden>
Date: Wed Sep 16 12:40:15 2015 +0300

    Return original interface mac

    In order to prevent nailgun get inconsistent network configuration in
    case of interface bonding mode return original mac address.

    Change-Id: Ib0a5c86484d86bf07098a51de87baae931db66ec
    Closes-Bug: #1496279
    (cherry picked from commit e623c89f94911c5bc70eb73abf77921c7a5e246c)

tags: removed: 70mu1-confirmed
Revision history for this message
Ksenia Svechnikova (kdemina) wrote :

Verified 7.0-mu-1

tags: removed: on-verification
tags: added: 7mu1-verified
Revision history for this message
Nastya Urlapova (aurlapova) wrote :

Moved to Fix Released according to a comment above.

tags: removed: 7mu1-verified
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.