Network verification fails with balance round-robin bonds

Bug #1546520 reported by Artem Panchenko
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Artem Roma
Mitaka
Fix Released
High
Vladimir Kuklin

Bug Description

Network verification fails if 'public' network is assigned to network bond in 'balance-rr' mode:

Verification failed.
Repo availability verification using public network failed on following nodes Untitled (be:be).
Following repos are not available - http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/
. Check your public network settings and availability of the repositories from public network. Please examine nailgun and astute logs for additional details.

When few network interfaces are bonded, network checker takes only one its slave while testing connectivity to repositories:

D, [2016-02-17T12:26:59.473908 #1712] DEBUG -- : net_probe.rb:87:in `block in <class:Net_probe>' about to execute: urlaccesscheck with setup -i enp0s5 -g 10.109.1.1 -a 10.109.1.2/24 --vlan 0 'http://10.109.0.2:8080/liberty-8.0/ubuntu/x86_64' 'http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/' 'http://archive.ubuntu.com/ubuntu/' 'http://10.109.0.2:8080/liberty-8.0/ubuntu/auxiliary'

That works for bonds in 'active-backup' mode, but in case of round robin mode ports on network switch are aggregated too, so incoming traffic is balanced between interfaces. That's why a lot of data is lost if you try to use only one slave NIC on target node (4 interfaces are bonded):

root@bootstrap:~# ip a add 10.109.1.2/24 dev enp0s5
root@bootstrap:~# ip link set enp0s5 up
root@bootstrap:~# ping 10.109.1.1
PING 10.109.1.1 (10.109.1.1) 56(84) bytes of data.
64 bytes from 10.109.1.1: icmp_seq=5 ttl=64 time=0.166 ms
64 bytes from 10.109.1.1: icmp_seq=9 ttl=64 time=0.186 ms
^C
--- 10.109.1.1 ping statistics ---
12 packets transmitted, 2 received, 83% packet loss, time 11072ms
rtt min/avg/max/mdev = 0.166/0.176/0.186/0.010 ms

I think we have to disable network checks for balance-rr bonds like we previously did for LACP (see bug #1376908).

Steps to reproduce:

1. Prepare ports on network switch (aggregate appropriate links, enable round-robin mode)
2. Create new environment, choose Neutron + VXLAN
3. Add 3 controller and 2 compute+ceph nodes
4. Configure network bonds for all nodes using 'balance-rr' mode, assign public network to them
5. Verify networks

Expected result: verification is passed
Actual: verification fails, repositories aren't accessible via public network

Revision history for this message
Artem Panchenko (apanchenko-8) wrote :
Changed in fuel:
status: New → Confirmed
Dmitry Pyzhov (dpyzhov)
tags: added: area-python team-network
tags: added: keep-in-9.0
Revision history for this message
Dmitry Pyzhov (dpyzhov) wrote :

Round-robin bonds is important configuration for our users. Changing priority to high because without working netchecker the only way to check feature health is by running manual deployment.

Changed in fuel:
importance: Medium → High
Revision history for this message
Aleksandr Didenko (adidenko) wrote :

Dmitry, network checker will not work for balance-rr bonds because in order to test them properly we need to configure them on bootstrap node (thus duplicate netconfig.pp task in bootstrap stage). So I agree with Artem, we should disable network checker for balance-rr bonds.

Artem Roma (aroma-x)
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Artem Roma (aroma-x)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/303981

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/303981
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=5c3ed6ef0f6636d4b6d274f179dacec27a370b3c
Submitter: Jenkins
Branch: master

commit 5c3ed6ef0f6636d4b6d274f179dacec27a370b3c
Author: Artem Roma <email address hidden>
Date: Mon Apr 11 12:10:45 2016 +0300

    Skip network verification for bonds in round robin mode

    Bond interfaces with 'balance-rr' mode are now skipped by network
    verification task for bootstrap nodes as network checker does not
    support this type of bonding.

    Change-Id: I39b2c5916fd6f39548af0e41d8e7f877983f73a2
    Closes-Bug: #1546520

Changed in fuel:
status: In Progress → Fix Committed
Artem Roma (aroma-x)
tags: added: move-to-mu
Changed in fuel:
status: Fix Committed → Confirmed
milestone: 9.0 → 9.0-updates
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
milestone: 9.1 → 9.2
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
milestone: 9.2 → 10.1
status: Confirmed → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/409816

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (stable/mitaka)

Reviewed: https://review.openstack.org/409816
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=a8375d5127943cb2cf069931875e593b3d8c4ac4
Submitter: Jenkins
Branch: stable/mitaka

commit a8375d5127943cb2cf069931875e593b3d8c4ac4
Author: Artem Roma <email address hidden>
Date: Mon Apr 11 12:10:45 2016 +0300

    Skip network verification for bonds in round robin mode

    Bond interfaces with 'balance-rr' mode are now skipped by network
    verification task for bootstrap nodes as network checker does not
    support this type of bonding.

    Change-Id: I39b2c5916fd6f39548af0e41d8e7f877983f73a2
    Closes-Bug: #1546520
    (cherry picked from commit 5c3ed6ef0f6636d4b6d274f179dacec27a370b3c)

Revision history for this message
Ekaterina Shutova (eshutova) wrote :

Network verification succeeded.
Verified on 9.2:
cat /etc/fuel_build_id:
 495
cat /etc/fuel_build_number:
 495
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-nailgun-9.0.0-1.mos8932.noarch
 network-checker-9.0.0-1.mos77.x86_64
 fuel-ostf-9.0.0-1.mos947.noarch
 fuel-ui-9.0.0-1.mos2855.noarch
 fuel-agent-9.0.0-1.mos291.noarch
 fuel-library9.0-9.0.0-1.mos8685.noarch
 nailgun-mcagents-9.0.0-1.mos785.noarch
 fuel-setup-9.0.0-1.mos6359.noarch
 shotgun-9.0.0-1.mos90.noarch
 python-fuelclient-9.0.0-1.mos364.noarch
 fuelmenu-9.0.0-1.mos276.noarch
 python-packetary-9.0.0-1.mos162.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8932.noarch
 fuel-notify-9.0.0-1.mos8685.noarch
 rubygem-astute-9.0.0-1.mos785.noarch
 fuel-release-9.0.0-1.mos6359.noarch
 fuel-mirror-9.0.0-1.mos162.noarch
 fuel-migrate-9.0.0-1.mos8685.noarch
 fuel-misc-9.0.0-1.mos8685.noarch
 fuel-bootstrap-cli-9.0.0-1.mos291.noarch
 fuel-openstack-metadata-9.0.0-1.mos8932.noarch
 fuel-utils-9.0.0-1.mos8685.noarch
 fuel-9.0.0-1.mos6359.noarch

tags: added: on-verification
tags: removed: on-verification
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.