Hi Hari,

I can recreate the problem. Once the setup has reached problematic state,
observed all the issues Amit has  mentioned.
But to reach this state I have to remove one slave interface from bond and
restart network services.

With normal flap of interface and network restart never able to reproduce
the problem(change of bond mac).  Tried running fab add_static_route
multiple  time. Which is original trigger.

Can we see the bonding configuration once ? Is it done through fab task ?

root@nodei9:~# ifconfig bond0
bond0     Link encap:Ethernet  HWaddr 00:25:90:e4:08:e5>>>>>>>>>>>>
          inet6 addr: fe80::225:90ff:fee4:8e5/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:66991876 errors:0 dropped:38 overruns:0 frame:0
          TX packets:40657426 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:14582541218 (14.5 GB)  TX bytes:15205158719 (15.2 GB)


root@nodei9:~# ifconfig vhost0
vhost0    Link encap:Ethernet  HWaddr 00:25:90:e4:08:e4>>>>>>>>>>>>> vhost
and bond having different config
          inet addr:192.168.22.4  Bcast:192.168.22.255  Mask:255.255.255.0
          inet6 addr: fe80::225:90ff:fee4:8e4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:10414086 errors:0 dropped:912 overruns:0 frame:0
          TX packets:10211823 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:13980201687 (13.9 GB)  TX bytes:17861939531 (17.8 GB)

In bonding with fab
‹‹‹‹‹‹‹‹‹‹------------------
root@nodei9:~# ifconfig p6p1
p6p1      Link encap:Ethernet  HWaddr 00:25:90:e4:08:e4 >>>>>>>>>>>>>>
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:53299835 errors:0 dropped:2425 overruns:0 frame:0
          TX packets:16579042 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:13166191099 (13.1 GB)  TX bytes:12684156223 (12.6 GB)


root@nodei9:~# ifconfig p6p2
p6p2      Link encap:Ethernet  HWaddr 00:25:90:e4:08:e4>>>>>>>>>>>>>>>
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:67287029 errors:0 dropped:2477 overruns:0 frame:0
          TX packets:40981312 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:14712314080 (14.7 GB)  TX bytes:15627041363 (15.6 GB)


root@nodei9:~# ifconfig bond0
bond0     Link encap:Ethernet  HWaddr 00:25:90:e4:08:e4>>>>>>>>>Both the
slave has mac of bond0
          inet6 addr: fe80::225:90ff:fee4:8e4/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:120613044 errors:0 dropped:4902 overruns:0 frame:0
          TX packets:57560635 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:27881390442 (27.8 GB)  TX bytes:28311340765 (28.3 GB)

root@nodei9:~# ifconfig p6p1 down
root@nodei9:~# ifconfig p6p1 bond0

root@nodei9:~# ifconfig bond0
bond0     Link encap:Ethernet  HWaddr 00:25:90:e4:08:e4 >>>> Even when we
shut the master interface bond mac is not changing
          inet6 addr: fe80::225:90ff:fee4:8e4/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:120744646 errors:0 dropped:4902 overruns:0 frame:0
          TX packets:57562048 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:27895888836 (27.8 GB)  TX bytes:28312065072 (28.3 GB)

Thanks and Regards,
Chhandak 











On 5/13/15, 3:04 AM, "amit surana" <email address hidden> wrote:

>Private bug reported:
>
>R2.20 build 13 Ubuntu 14.0.4.
>
>
>Flapping a bond interface can cause its MAC address to change. Other
>services that rely on this MAC address, should also be restarted so
>that they can attach to the new MAC address. Details below.
>
>
>Consider a case where the controller/compute node is connected to the IP
>fabric via a 2 interface bond (LAG). The bond assumes the mac address of
>one of the slave interfaces (usually the first one that comes up).
>
>On the compute, the vhost0 interface also gets assigned the same MAC
>address as that of the bond interface. This MAC address is also used by
>keepalived/haproxy as the MAC address corresponding to the VIP.
>
>Now, if the bond interface flaps (networking service was restarted for
>instance) and the MAC address of the bond interface changes to that of a
>different slave interface, it is seen that vhost0 interface still points
>to the old MAC and this breaks all the connectivity to the compute. As
>far as the KA/HAproxy is concerned, though the services are running, the
>VIP isn't owned by any of the nodes, and so this functionality also
>breaks.
>
>The above scenario was simulated on the solution testbed by having the
>fab script add a static route to all the nodes on a working cluster.
>After adding the static route the fab add_static_route script restarts
>the networking service;  this flaps the bond interface and causes the
>MAC address of the bond interface to change, which in turn leads to the
>above noted issues.
>
>If vrouter/keepalive services are restarted, the issue is resolved.
>
>
>bond0     Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8a
>em1       Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8a
>em2       Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8a
>vhost0    Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8a
>
>==restart networking==
>
>bond0     Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8b
>em1       Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8b
>em2       Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8b
>vhost0    Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8a
>
>** Affects: juniperopenstack
>     Importance: High
>         Status: New
>
>** Affects: juniperopenstack/r2.20
>     Importance: High
>         Status: New
>
>** Affects: juniperopenstack/trunk
>     Importance: High
>         Status: New
>
>** Also affects: juniperopenstack/r2.20
>   Importance: Undecided
>       Status: New
>
>** Also affects: juniperopenstack/trunk
>   Importance: High
>       Status: New
>
>** Changed in: juniperopenstack/r2.20
>   Importance: Undecided => High
>
>** Changed in: juniperopenstack/r2.20
>    Milestone: None => r2.20-fcs
>
>** Changed in: juniperopenstack/trunk
>    Milestone: r2.20-fcs => r2.30-fcs
>
>** Description changed:
>
>- Consider a case where the controller/compute node is connected to the IP
>- fabric via a 2 interface bond (LAG). The bond assumes the mac address of
>- one of the slave interfaces (usually the first one that comes up).
>+ R2.20 build 13 Ubuntu 14.0.4.
>+ 
>+ 
>+ Flapping a bond interface can cause its MAC address to change. Other
>+ services that rely on this MAC address, should also be restarted so
>+ that they can attach to the new MAC address. Details below.
>+ 
>+ 
>+ Consider a case where the controller/compute node is connected to the
>IP fabric via a 2 interface bond (LAG). The bond assumes the mac address
>of one of the slave interfaces (usually the first one that comes up).
>  
>  On the compute, the vhost0 interface also gets assigned the same MAC
>  address as that of the bond interface. This MAC address is also used by
>  keepalived/haproxy as the MAC address corresponding to the VIP.
>  
>  Now, if the bond interface flaps (networking service was restarted for
>  instance) and the MAC address of the bond interface changes to that of a
>  different slave interface, it is seen that vhost0 interface still points
>  to the old MAC and this breaks all the connectivity to the compute. As
>  far as the KA/HAproxy is concerned, though the services are running, the
>  VIP isn't owned by any of the nodes, and so this functionality also
>  breaks.
>  
>  The above scenario was simulated on the solution testbed by having the
>  fab script add a static route to all the nodes on a working cluster.
>  After adding the static route the fab add_static_route script restarts
>- the networking service;  this flaps the bond interface and caused the
>- MAC address of the bond interface to change, which in turn lead to the
>+ the networking service;  this flaps the bond interface and causes the
>+ MAC address of the bond interface to change, which in turn leads to the
>  above noted issues.
>  
>- If vrouter/keepalive services are restarted, the issue is resolved -
>- which points towards a potential solution.
>+ If vrouter/keepalive services are restarted, the issue is resolved.
>  
>  
>- bond0     Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8a
>- em1       Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8a
>- em2       Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8a
>- vhost0    Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8a
>+ bond0     Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8a
>+ em1       Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8a
>+ em2       Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8a
>+ vhost0    Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8a
>  
>  ==restart networking==
>  
>  bond0     Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8b
>- em1       Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8b
>- em2       Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8b
>+ em1       Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8b
>+ em2       Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8b
>  vhost0    Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8a
>
>-- 
>You received this bug notification because you are a member of Contrail
>Systems engineering, which is subscribed to Juniper Openstack.
>https://bugs.launchpad.net/bugs/1454420
>
>Title:
>  [R2.20] Bond interface flap can break haproxy/vrouter networking
>
>Status in Juniper Openstack distribution:
>  New
>Status in Juniper Openstack r2.20 series:
>  New
>Status in Juniper Openstack trunk series:
>  New
>
>Bug description:
>  R2.20 build 13 Ubuntu 14.0.4.
>
>  
>  Flapping a bond interface can cause its MAC address to change. Other
>  services that rely on this MAC address, should also be restarted so
>  that they can attach to the new MAC address. Details below.
>
>  
>  Consider a case where the controller/compute node is connected to the
>IP fabric via a 2 interface bond (LAG). The bond assumes the mac address
>of one of the slave interfaces (usually the first one that comes up).
>
>  On the compute, the vhost0 interface also gets assigned the same MAC
>  address as that of the bond interface. This MAC address is also used
>  by keepalived/haproxy as the MAC address corresponding to the VIP.
>
>  Now, if the bond interface flaps (networking service was restarted for
>  instance) and the MAC address of the bond interface changes to that of
>  a different slave interface, it is seen that vhost0 interface still
>  points to the old MAC and this breaks all the connectivity to the
>  compute. As far as the KA/HAproxy is concerned, though the services
>  are running, the VIP isn't owned by any of the nodes, and so this
>  functionality also breaks.
>
>  The above scenario was simulated on the solution testbed by having the
>  fab script add a static route to all the nodes on a working cluster.
>  After adding the static route the fab add_static_route script restarts
>  the networking service;  this flaps the bond interface and causes the
>  MAC address of the bond interface to change, which in turn leads to
>  the above noted issues.
>
>  If vrouter/keepalive services are restarted, the issue is resolved.
>
>  
>  bond0     Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8a
>  em1       Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8a
>  em2       Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8a
>  vhost0    Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8a
>
>  ==restart networking==
>
>  bond0     Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8b
>  em1       Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8b
>  em2       Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8b
>  vhost0    Link encap:Ethernet  HWaddr 08:9e:01:d9:27:8a
>
>To manage notifications about this bug go to:
>https://bugs.launchpad.net/juniperopenstack/+bug/1454420/+subscriptions