| 2015-06-29 14:44:42 |
Andrey Danin |
bug |
|
|
added bug |
| 2015-06-29 14:47:20 |
Andrey Danin |
attachment added |
|
diag.tar.gz https://bugs.launchpad.net/fuel/+bug/1469746/+attachment/4421967/+files/diag.tar.gz |
|
| 2015-06-29 16:03:39 |
Andrey Danin |
description |
Affects 6.1 GA.
A node should have two LACP bonds set up, but the second one has a type round-robin instead 802.3ad.
Bonds ovs-bond0 and ovs-bond1 (Linux bonds despite their names) are configured in the same way as LACP Linux bonds with xmit_hash_policy=layer3+4 and rate=fast. After the deploy is done (with error because of lack of connectivity) ovs-bond0 is configured properly but ovs-bond1 is configured as round-robin non LACP. However in /etc/network/interfaces.d their configurations almost equal with the only difference in "post-up sleep 45" for ovs-bond1. |
Affects 6.1 GA.
Ubuntu, Neutron, linux bonds.
A node should have two LACP bonds set up, but the second one has a type round-robin instead 802.3ad.
Bonds ovs-bond0 and ovs-bond1 (Linux bonds despite their names) are configured in the same way as LACP Linux bonds with xmit_hash_policy=layer3+4 and rate=fast. After the deploy is done (with error because of lack of connectivity) ovs-bond0 is configured properly but ovs-bond1 is configured as round-robin non LACP. However in /etc/network/interfaces.d their configurations almost equal with the only difference in "post-up sleep 45" for ovs-bond1.
The problem repeats on each node with bonds in my env. |
|
| 2015-06-29 21:10:50 |
Mike Scherbakov |
nominated for series |
|
fuel/6.1.x |
|
| 2015-06-29 21:10:50 |
Mike Scherbakov |
bug task added |
|
fuel/6.1.x |
|
| 2015-06-29 21:11:00 |
Mike Scherbakov |
fuel/6.1.x: milestone |
|
6.1-updates |
|
| 2015-06-29 21:11:14 |
Mike Scherbakov |
tags |
l23network |
customer-found l23network |
|
| 2015-06-29 21:11:28 |
Mike Scherbakov |
bug |
|
|
added subscriber Andrew Woodward |
| 2015-06-30 13:14:57 |
Oleksiy Molchanov |
fuel/6.1.x: importance |
Undecided |
High |
|
| 2015-06-30 13:15:07 |
Oleksiy Molchanov |
fuel/6.1.x: assignee |
|
Sergey Vasilenko (xenolog) |
|
| 2015-06-30 13:15:11 |
Oleksiy Molchanov |
fuel: status |
New |
Confirmed |
|
| 2015-06-30 13:15:14 |
Oleksiy Molchanov |
fuel/6.1.x: status |
New |
Confirmed |
|
| 2015-06-30 13:52:31 |
Oleksiy Molchanov |
fuel: status |
Confirmed |
Incomplete |
|
| 2015-06-30 13:52:34 |
Oleksiy Molchanov |
fuel/6.1.x: status |
Confirmed |
Incomplete |
|
| 2015-07-01 19:36:41 |
Andrey Danin |
fuel: status |
Incomplete |
Confirmed |
|
| 2015-07-01 19:36:45 |
Andrey Danin |
fuel/6.1.x: status |
Incomplete |
Confirmed |
|
| 2015-08-04 11:46:26 |
Sergey Vasilenko |
fuel: status |
Confirmed |
Triaged |
|
| 2015-08-04 14:15:25 |
OpenStack Infra |
fuel: status |
Triaged |
In Progress |
|
| 2015-08-04 14:24:18 |
Sergey Vasilenko |
fuel/6.1.x: status |
Confirmed |
In Progress |
|
| 2015-08-05 07:57:19 |
OpenStack Infra |
fuel: status |
In Progress |
Fix Committed |
|
| 2015-09-02 15:51:13 |
Stanislav Makar |
fuel/6.1.x: assignee |
Sergey Vasilenko (xenolog) |
Stanislav Makar (smakar) |
|
| 2015-09-02 21:30:17 |
Andrey Danin |
fuel: status |
Fix Committed |
Confirmed |
|
| 2015-09-02 22:19:46 |
Andrey Danin |
description |
Affects 6.1 GA.
Ubuntu, Neutron, linux bonds.
A node should have two LACP bonds set up, but the second one has a type round-robin instead 802.3ad.
Bonds ovs-bond0 and ovs-bond1 (Linux bonds despite their names) are configured in the same way as LACP Linux bonds with xmit_hash_policy=layer3+4 and rate=fast. After the deploy is done (with error because of lack of connectivity) ovs-bond0 is configured properly but ovs-bond1 is configured as round-robin non LACP. However in /etc/network/interfaces.d their configurations almost equal with the only difference in "post-up sleep 45" for ovs-bond1.
The problem repeats on each node with bonds in my env. |
Affects 6.1 GA.
Ubuntu, Neutron, linux bonds.
A node should have two LACP bonds set up, but the second one has a type round-robin instead 802.3ad.
Steps to reproduce:
1) Create a new env.
2) Add a node with 5 NICs - 1 for PXE and 4 others form two LACP bonds with 2 NICs in each (an appropriate setup on the switch side should be done). In my case I added 3 such nodes with Ceph role assigned.
3) Assign Public, Management to bond0 and Storage, Private to bond1. Using API call mark bonds as linux (not OVS), mode=802.3ad, xmit_hash_policy=layer3+4, and rate=fast (the code I use to do this http://paste.openstack.org/show/442567/)
4) Add other nodes required for deployment.
5) Deploy the env.
Expected result:
he deployment should pass.
All bonds on Ceph nodes should be configured as 802.3ad with rate=fast.
Actual result:
The result is intermittent. For 3 such Ceph nodes I have 80% chance of broken env.
The deployment fails while executing /etc/puppet/modules/osnailyfacter/modular/astute/ceph_ready_check.rb because of lack of connectivity through the bonds.
One or more nodes have the second bond (bond1) configured in round-robin mode.
root@node-20:~# grep 'Bonding Mode' /proc/net/bonding/bond*
/proc/net/bonding/bond0:Bonding Mode: IEEE 802.3ad Dynamic link aggregation
/proc/net/bonding/bond1:Bonding Mode: load balancing (round-robin)
Even if a bond is configured in 802.3ad mode it always have lacp_rate=slow.
However in /etc/network/interfaces.d bond configurations are correct and almost equal with the only difference in "post-up sleep 45" for bond1.
(See http://paste.openstack.org/show/442570/ for real example). If a bond is restarted (ifdown -a;ifup -a) it gets configured in the right way.
Workaround:
If a deployment fails go to Ceph nodes, restart broken bonds (ifdown bond1;ifup eth2 eth3; # If I run ifup bond1 it freezes forever, so I have to run ifup <nics-belonged-to-bond1>, see http://paste.openstack.org/show/442583/), check that ceph osds are ok, then run deploy again.
So, I see three bugs here:
1) lacp_rate always slow for runtime config right after deployment.
2) sometimes the second bond doesn't get the right mode.
3) It's not possible to bring bonds up by ifup command. |
|
| 2015-09-03 11:52:46 |
Stanislav Makar |
fuel: assignee |
Sergey Vasilenko (xenolog) |
Stanislav Makar (smakar) |
|
| 2015-09-03 11:52:52 |
Stanislav Makar |
fuel: status |
Confirmed |
In Progress |
|
| 2015-09-04 19:46:38 |
OpenStack Infra |
fuel: assignee |
Stanislav Makar (smakar) |
Sergey Vasilenko (xenolog) |
|
| 2015-09-04 19:46:46 |
OpenStack Infra |
fuel: assignee |
Sergey Vasilenko (xenolog) |
Stanislav Makar (smakar) |
|
| 2015-09-04 20:31:05 |
OpenStack Infra |
fuel: assignee |
Stanislav Makar (smakar) |
Sergey Vasilenko (xenolog) |
|
| 2015-09-05 12:05:42 |
OpenStack Infra |
fuel: status |
In Progress |
Fix Committed |
|
| 2015-09-06 18:48:42 |
Vladimir Kuklin |
fuel: status |
Fix Committed |
Triaged |
|
| 2015-09-06 18:57:47 |
OpenStack Infra |
fuel: status |
Triaged |
In Progress |
|
| 2015-09-06 18:57:47 |
OpenStack Infra |
fuel: assignee |
Sergey Vasilenko (xenolog) |
Vladimir Kuklin (vkuklin) |
|
| 2015-09-06 19:03:32 |
OpenStack Infra |
fuel: assignee |
Vladimir Kuklin (vkuklin) |
Sergey Vasilenko (xenolog) |
|
| 2015-09-07 20:55:14 |
OpenStack Infra |
fuel: status |
In Progress |
Fix Committed |
|
| 2015-09-07 21:22:18 |
Sergii Golovatiuk |
nominated for series |
|
fuel/8.0.x |
|
| 2015-09-07 21:22:18 |
Sergii Golovatiuk |
bug task added |
|
fuel/8.0.x |
|
| 2015-09-07 21:22:25 |
Sergii Golovatiuk |
fuel/8.0.x: status |
New |
Fix Committed |
|
| 2015-09-07 21:22:28 |
Sergii Golovatiuk |
fuel/8.0.x: importance |
Undecided |
High |
|
| 2015-09-07 21:22:36 |
Sergii Golovatiuk |
fuel/8.0.x: assignee |
|
Sergey Vasilenko (xenolog) |
|
| 2015-09-07 21:22:41 |
Sergii Golovatiuk |
fuel/8.0.x: milestone |
|
8.0 |
|
| 2015-09-11 09:23:37 |
Stanislav Makar |
tags |
customer-found l23network |
customer-found l23network on-verification |
|
| 2015-09-11 12:20:18 |
Stanislav Makar |
fuel/8.0.x: status |
Fix Committed |
Fix Released |
|
| 2015-09-11 12:20:26 |
Stanislav Makar |
tags |
customer-found l23network on-verification |
customer-found l23network |
|
| 2015-09-18 11:19:27 |
Vladimir Kuklin |
fuel/6.1.x: status |
In Progress |
Fix Committed |
|
| 2015-10-06 13:00:33 |
Matthew Mosesohn |
nominated for series |
|
fuel/7.0.x |
|
| 2015-10-06 13:00:33 |
Matthew Mosesohn |
bug task added |
|
fuel/7.0.x |
|
| 2015-10-06 13:00:47 |
Matthew Mosesohn |
fuel/7.0.x: assignee |
|
Sergii Golovatiuk (sgolovatiuk) |
|
| 2015-10-06 13:00:49 |
Matthew Mosesohn |
fuel/7.0.x: milestone |
|
7.0-mu-1 |
|
| 2015-10-06 13:00:51 |
Matthew Mosesohn |
fuel/7.0.x: importance |
Undecided |
High |
|
| 2015-10-06 13:00:53 |
Matthew Mosesohn |
fuel/7.0.x: status |
New |
Fix Committed |
|
| 2015-10-06 17:46:08 |
Dmitry Pyzhov |
fuel/7.0.x: milestone |
7.0-mu-1 |
7.0-updates |
|
| 2015-10-22 03:15:15 |
Dmitry Pyzhov |
tags |
customer-found l23network |
area-library customer-found l23network |
|
| 2015-11-30 15:23:59 |
Dmitry Pyzhov |
fuel: milestone |
7.0 |
8.0 |
|
