[Ubuntu18.04][Witherspoon DD2.2] netplan fails to configure network bridge interface (tg3/BCM5719)

Bug #1759056 reported by bugproxy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
The Ubuntu-power-systems project
Fix Released
High
Canonical Foundations Team
netplan.io (Ubuntu)
Fix Released
High
Canonical Foundations Team

Bug Description

Problem:
-----------
Network bridge interface setup fails with netplan configuration

br0: port 1(enP52p1s0f0) entered blocking state
br0: port 1(enP52p1s0f0) entered disabled state
device enP52p1s0f0 entered promiscuous mode
br0: port 1(enP52p1s0f0) entered blocking state
br0: port 1(enP52p1s0f0) entered forwarding state
IPv6: ADDRCONF(NETDEV_UP): br0: link is not ready
br0: port 1(enP52p1s0f0) entered disabled state
device enP52p1s0f0 left promiscuous mode
br0: port 1(enP52p1s0f0) entered disabled state

Machine Type: Witherspoon BMC baremetal DD2.2
uname -a
Linux ltciofvtr-spoon1 4.15.0-12-generic #13-Ubuntu SMP Wed Mar 7 21:37:03 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

package:
netplan.io 0.34

lspci -nn | grep 0034:01:00.0
0034:00:00.0 PCI bridge [0604]: IBM Device [1014:04c1]
0034:01:00.0 Ethernet controller [0200]: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)

# ethtool -i enP52p1s0f0
driver: tg3
version: 3.137
firmware-version: 5719-v1.38i
expansion-rom-version:
bus-info: 0034:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

t# ifconfig enP52p1s0f0
enP52p1s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 65.65.65.32 netmask 255.255.255.0 broadcast 65.65.65.255
        ether 98:be:94:01:02:80 txqueuelen 1000 (Ethernet)
        RX packets 81026 bytes 6025544 (6.0 MB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 4946 bytes 431946 (431.9 KB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
        device interrupt 76

- - -Steps to recreate - - -
1. Edit /etc/netplan/01-netcfg.yaml and add bridge interface entry br0

# cat /etc/netplan/01-netcfg.yaml
# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
  version: 2
  renderer: networkd
  ethernets:
    enP5p1s0f0:
      addresses: [ 9.xx.xx.xx/20 ]
      gateway4: 9.xx.xx.xx
      nameservers:
          search: [ xx.xx.ibm.com ]
          addresses:
              - 9.xx.xx.x
    enP52p1s0f0:
      dhcp4: no
      dhcp6: no
  bridges:
    br0:
      interfaces: [ enP52p1s0f0 ]
      addresses: [ 65.65.65.32/24 ]
      dhcp4: no

2. Apply the changes by running below command
# netplan --debug apply
** (generate:26009): DEBUG: 01:41:49.588: Processing input file //etc/netplan/01-netcfg.yaml..
** (generate:26009): DEBUG: 01:41:49.588: starting new processing pass
** (generate:26009): DEBUG: 01:41:49.588: enP52p1s0f0: setting default backend to 1
** (generate:26009): DEBUG: 01:41:49.588: br0: setting default backend to 1
** (generate:26009): DEBUG: 01:41:49.588: enP5p1s0f0: setting default backend to 1
** (generate:26009): DEBUG: 01:41:49.588: Generating output files..
** (generate:26009): DEBUG: 01:41:49.588: NetworkManager: definition enP52p1s0f0 is not for us (backend 1)
** (generate:26009): DEBUG: 01:41:49.588: NetworkManager: definition br0 is not for us (backend 1)
** (generate:26009): DEBUG: 01:41:49.588: NetworkManager: definition enP5p1s0f0 is not for us (backend 1)
DEBUG:netplan generated networkd configuration exists, restarting networkd
DEBUG:no netplan generated NM configuration exists
DEBUG:device enP52p1s0f1 operstate is up, not replugging
DEBUG:netplan triggering .link rules for enP52p1s0f1
DEBUG:device enP5p1s0f0 operstate is up, not replugging
DEBUG:netplan triggering .link rules for enP5p1s0f0
DEBUG:device lo operstate is unknown, not replugging
DEBUG:netplan triggering .link rules for lo
DEBUG:replug enP52p1s0f2: unbinding 0034:01:00.2 from /sys/bus/pci/drivers/tg3
DEBUG:replug enP52p1s0f2: rebinding 0034:01:00.2 to /sys/bus/pci/drivers/tg3
DEBUG:replug enP5p1s0f1: unbinding 0005:01:00.1 from /sys/bus/pci/drivers/tg3
DEBUG:replug enP5p1s0f1: rebinding 0005:01:00.1 to /sys/bus/pci/drivers/tg3
DEBUG:device sit0 operstate is unknown, not replugging
DEBUG:netplan triggering .link rules for sit0
DEBUG:device enP52p1s0f0 operstate is up, not replugging
DEBUG:netplan triggering .link rules for enP52p1s0f0
DEBUG:replug enP52p1s0f3: unbinding 0034:01:00.3 from /sys/bus/pci/drivers/tg3
DEBUG:replug enP52p1s0f3: rebinding 0034:01:00.3 to /sys/bus/pci/drivers/tg3

$ ifconfig -a | grep br0

It has failed to configured bridge interface.

dmesg:
tg3 0034:01:00.2: Using 64-bit DMA iommu bypass
tg3 0034:01:00.2 eth0: Tigon3 [partno(00E2872) rev 5719001] (PCI Express) MAC address 98:be:94:01:02:82
tg3 0034:01:00.2 eth0: attached PHY is 5719C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
tg3 0034:01:00.2 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
tg3 0034:01:00.2 eth0: dma_rwctrl[00000000] dma_mask[64-bit]
tg3 0034:01:00.2 enP52p1s0f2: renamed from eth0
tg3 0005:01:00.1: Using 64-bit DMA iommu bypass
tg3 0005:01:00.1 eth0: Tigon3 [partno(BCM95719) rev 5719001] (PCI Express) MAC address 70:e2:84:14:29:fb
tg3 0005:01:00.1 eth0: attached PHY is 5719C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
tg3 0005:01:00.1 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
tg3 0005:01:00.1 eth0: dma_rwctrl[00000000] dma_mask[64-bit]
tg3 0005:01:00.1 enP5p1s0f1: renamed from eth0
IPv6: ADDRCONF(NETDEV_UP): enP52p1s0f2: link is not ready
tg3 0005:01:00.1 enP5p1s0f1: Link is up at 1000 Mbps, full duplex
tg3 0005:01:00.1 enP5p1s0f1: Flow control is on for TX and on for RX
tg3 0005:01:00.1 enP5p1s0f1: EEE is disabled
net_ratelimit: 5 callbacks suppressed
IPv4: martian source 255.255.255.255 from 9.47.72.146, on dev enP5p1s0f1
ll header: 00000000: ff ff ff ff ff ff ea e3 8e 00 2d 02 08 00 ..........-...
IPv6: ADDRCONF(NETDEV_UP): enP5p1s0f1: link is not ready
IPv6: ADDRCONF(NETDEV_CHANGE): enP5p1s0f1: link becomes ready
tg3 0034:01:00.3: Using 64-bit DMA iommu bypass
tg3 0034:01:00.3 eth0: Tigon3 [partno(00E2872) rev 5719001] (PCI Express) MAC address 98:be:94:01:02:83
tg3 0034:01:00.3 eth0: attached PHY is 5719C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
tg3 0034:01:00.3 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
tg3 0034:01:00.3 eth0: dma_rwctrl[00000000] dma_mask[64-bit]
tg3 0034:01:00.3 enP52p1s0f3: renamed from eth0
br0: port 1(enP52p1s0f0) entered blocking state
br0: port 1(enP52p1s0f0) entered disabled state
device enP52p1s0f0 entered promiscuous mode
device enP52p1s0f0 left promiscuous mode
br0: port 1(enP52p1s0f0) entered disabled state
IPv6: ADDRCONF(NETDEV_UP): enP52p1s0f3: link is not ready

--- expected ---
bridge interface br0 should come online

Guys I tried and failed using netplan to configure a bridge.
We better let canonical answer us how to proceed here.

bugproxy (bugproxy)
tags: added: architecture-ppc64le bugnameltc-165777 severity-high targetmilestone-inin1804
Changed in ubuntu:
assignee: nobody → Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage)
affects: ubuntu → plan (Ubuntu)
Luciano Chavez (lnx1138)
affects: plan (Ubuntu) → netplan.io (Ubuntu)
Frank Heimes (fheimes)
tags: added: triage-g
Changed in ubuntu-power-systems:
importance: Undecided → High
assignee: nobody → Canonical Foundations Team (canonical-foundations)
Manoj Iyer (manjo)
Changed in netplan.io (Ubuntu):
assignee: Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) → Canonical Foundations Team (canonical-foundations)
importance: Undecided → High
Revision history for this message
Steve Langasek (vorlon) wrote :

> DEBUG:device enP52p1s0f0 operstate is up, not replugging

Your ifconfig shows that this interface is up and has a configured IP address. Do I understand correctly that you encountered this error when trying to move the IP from the physical interface to a bridge?

Changed in ubuntu-power-systems:
status: New → Triaged
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2018-04-12 03:05 EDT-------
(In reply to comment #9)
> > DEBUG:device enP52p1s0f0 operstate is up, not replugging
>
> Your ifconfig shows that this interface is up and has a configured IP
> address. Do I understand correctly that you encountered this error when
> trying to move the IP from the physical interface to a bridge?

Yes, netplan is not flushing the IP addr of enP52p1s0f0 when moved to br0

but when machine is rebooted I am able to see br0 created.

Steve Langasek (vorlon)
Changed in netplan.io (Ubuntu):
status: New → Triaged
Revision history for this message
Patricia Gaughen (gaughen) wrote :

This did not make the 18.04 release, after the release has GAd, we will dive into this one deeper and evaluate for delivering for 18.04.1

tags: added: id-5adfc2aa2053897c8d9c68bb
Frank Heimes (fheimes)
tags: added: triage-a
removed: triage-g
tags: added: ubuntu-18.04.1
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-06-12 01:49 EDT-------
please advice if this issue is being considered in ubuntu 18.04.01?

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

This issue was reported prior to 18.04 GA, using an older version of netplan.io. Have you tested the same with a newer version of netplan?

It seems to work just fine here -- with two network interfaces connected to the system, I connect using ens6; then modify the netplan configuration to move the IP to a newly created bridge (added to the config -- not separately created from the command-line), then run 'sudo netplan apply'.

The bridge interface is created and the IP is moved to it, and I do not lose connectivity.

On reboot, the system comes up normally, and I have connectivity over br0.

I see no reason for this to behave any differently on tg3, but I've managed to get access to a system with such an interface, so I'll test that as well.

Steve Langasek (vorlon)
Changed in netplan.io (Ubuntu):
status: Triaged → Incomplete
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

ubuntu@AcerAspire:~$ sudo ethtool -i enp2s0
driver: tg3
version: 3.137
firmware-version: sb
expansion-rom-version:
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

I am unable to reproduce this with tg3 either.

Could you please share the exact contents of the netplan YAML intended to configure the bridge, along with the contents of /run/systemd/networkd?

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-06-22 08:46 EDT-------
(In reply to comment #13)
> This issue was reported prior to 18.04 GA, using an older version of
> netplan.io. Have you tested the same with a newer version of netplan?
>
> It seems to work just fine here -- with two network interfaces connected to
> the system, I connect using ens6; then modify the netplan configuration to
> move the IP to a newly created bridge (added to the config -- not separately
> created from the command-line), then run 'sudo netplan apply'.
>
> The bridge interface is created and the IP is moved to it, and I do not lose
> connectivity.
>
> On reboot, the system comes up normally, and I have connectivity over br0.
>
> I see no reason for this to behave any differently on tg3, but I've managed
> to get access to a system with such an interface, so I'll test that as well.

the bug was reported for 0.34 version, where it failed to configure bridge and with the latest version of netplan.io 0.36.2, bridge interface is configured. and after reboot the connectivity for br0 comes up.

We can close this bug as fixed

Revision history for this message
Steve Langasek (vorlon) wrote :

Thanks for confirming. Closing the LP bug.

Changed in netplan.io (Ubuntu):
status: Incomplete → Fix Released
Changed in ubuntu-power-systems:
status: Triaged → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-06-25 02:00 EDT-------
Perhaps the bridge interface is not pingable after configuration in yaml and run 'netplan apply' command.

It requires a machine reboot for the bridge interface to be pingable, I will be opening a separate bug for this.

Revision history for this message
bugproxy (bugproxy) wrote :
Download full text (9.2 KiB)

------- Comment From <email address hidden> 2018-07-23 06:03 EDT-------
Configuring Bridge with latest version of netplan is failing again, even after reboot of server. I could see the same error messages that are initially reported in the BUG.

I had following entry in netplan yaml and rebooted the server and could not the bridge interface under ifconfig output.

root@ltciofvtr-bostonlc1:~# cat /etc/net
netconfig netplan/ netscript/ network/ networks
root@ltciofvtr-bostonlc1:~# cat /etc/netplan/01-netcfg.yaml
# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
version: 2
renderer: networkd
ethernets:
enP2p1s0f2:
addresses: [ 9.47.78.174/20 ]
gateway4: 9.47.79.254
nameservers:
search: [ pok.stglabs.ibm.com ]
addresses:
- "9.3.1.200"
enP48p1s0f0:
dhcp4: no
addresses: [ 14.14.14.14/24 ]
enP48p1s0f1:
dhcp4: no
addresses: [ 15.15.15.15/24 ]
enP51p1s0f0:
dhcp4: no
optional: true
enP51p1s0f1:
dhcp4: no
optional: true
enP52p1s0f1:
dhcp4: no
optional: true

bridges:
br0:
interfaces: [ enP52p1s0f1 ]
addresses: [ 201.1.1.174/24 ]
dhcp4: no
# bonds:
# bond14:
# dhcp4: no
# interfaces: [ enP51p1s0f0, enP51p1s0f1 ]
# parameters:
# mode: balance-xor
# primary: enP51p1s0f0
# mii-monitor-interval: 100
# addresses: [ 199.1.1.174/24 ]
root@ltciofvtr-bostonlc1:~#
root@ltciofvtr-bostonlc1:~# ifconfig br0
br0: error fetching interface information: Device not found
root@ltciofvtr-bostonlc1:~#
root@ltciofvtr-bostonlc1:~# ifconfig enP52p1s0f1
enP52p1s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 24:8a:07:b4:e5:03 txqueuelen 1000 (Ethernet)
RX packets 3 bytes 420 (420.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

root@ltciofvtr-bostonlc1:~#

dmessages
=================

root@ltciofvtr-bostonlc1:~# dmesg | grep br0
[ 12.812776] IPv6: ADDRCONF(NETDEV_UP): br0: link is not ready
[ 12.813665] br0: port 1(enP52p1s0f1) entered blocking state
[ 12.813666] br0: port 1(enP52p1s0f1) entered disabled state
[ 13.835419] br0: port 1(enP52p1s0f1) entered blocking state
[ 13.835421] br0: port 1(enP52p1s0f1) entered forwarding state
[ 13.835515] IPv6: ADDRCONF(NETDEV_CHANGE): br0: link becomes ready
[ 17.363001] br0: port 1(enP52p1s0f1) entered disabled state
[ 17.364000] br0: port 1(enP52p1s0f1) entered disabled state
[ 118.588264] virbr0: port 1(virbr0-nic) entered blocking state
[ 118.588268] virbr0: port 1(virbr0-nic) entered disabled state
[ 118.588340] device virbr0-nic entered promiscuous mode
[ 118.840087] device virbr0-nic left promiscuous mode
[ 118.840196] virbr0: port 1(virbr0-nic) entered disabled state
root@ltciofvtr-bostonlc1:~#
root@ltciofvtr-bostonlc1:~#
root@ltciofvtr-bostonlc1:~#
root@ltciofvtr-bostonlc1:~# netplan --debug apply
** (generate:8844): DEBUG: 05:54:28.780: Processing input file //etc/netplan/01-netcfg.yaml..
** (generate:8844): DEBUG: 05:54:28.780: starting new processing pass
** (generate:8844): DEBUG: 05:54:28.780: enP48p1s0f1: setting default backend to 1
** (generate:8844): DEBUG: 05:54:...

Read more...

Revision history for this message
Andrew Cloke (andrew-cloke) wrote :

This Launchpad bug has now been closed. If you are still experiencing netplan issues, could you raise a new bug, referencing this one if necessary?
Thanks.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.