Ubuntu
linux package

Network device on z/VM instance incorrectly reports line speed

Bug #1667828 reported by Jeff Lane  on 2017-02-24

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Ubuntu on IBM z Systems	Fix Released	Undecided	Frank Heimes
	linux (Ubuntu)	Fix Released	Undecided	Unassigned

Bug Description

Line speed is incorrectly reported on z/VM instances in /sys when compared to what is claimed to be supported and advertised.

for example, ethtool shows supported and advertised speeds of 1000Mb/s but /sys is only indicating 10Mb/s.

ubuntu@hwe0008:~$ ethtool enc600
Settings for enc600:
        Supported ports: [ TP ]
        Supported link modes: 10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Advertised link modes: 10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Speed: 10Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: Unknown
Cannot get wake-on-lan settings: Operation not permitted
        Link detected: yes
ubuntu@hwe0008:~$ cat /sys/class/net/enc600/speed
10

I'm presuming this issue lies somewhere in the qeth driver:
P: /devices/qeth/0.0.0600/net/enc600
E: DEVPATH=/devices/qeth/0.0.0600/net/enc600
E: ID_NET_DRIVER=qeth_l2
E: ID_NET_LINK_FILE=/lib/systemd/network/99-default.link
E: ID_NET_NAME=enc600
E: ID_NET_NAME_MAC=enx02280b000004
E: ID_NET_NAME_PATH=enc600
E: ID_PATH=ccwgroup-0.0.0600
E: ID_PATH_TAG=ccwgroup-0_0_0600
E: IFINDEX=2
E: INTERFACE=enc600
E: SUBSYSTEM=net
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/enc600 /sys/subsystem/net/devices/enc600
E: TAGS=:systemd:
E: USEC_INITIALIZED=3784115

This is not an issue on my LPAR that uses the same drivers (but also adds ctcm and zfcp, so maybe therein lies the difference in speed reporting?)

In any case, if the speeds cannot be correctly reported, they shouldn't be reported at all. virtio handles this by setting speed to -1:
ubuntu@s1lp9g003:~$ cat /sys/class/net/eth0/speed
-1
ubuntu@s1lp9g003:~$ ethtool eth0
Settings for eth0:
Supported ports: [ ]
Supported link modes: Not reported
Supported pause frame use: No
Supports auto-negotiation: No
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Speed: Unknown!
Duplex: Unknown! (255)
Port: Other
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Cannot get wake-on-lan settings: Operation not permitted
Link detected: yes

That last example is a 1Gb/s interface on a z/KVM instance.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.8.0-36-generic 4.8.0-36.36~16.04.1
ProcVersionSignature: Ubuntu 4.8.0-36.36~16.04.1-generic 4.8.11
Uname: Linux 4.8.0-36-generic s390x
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: s390x
Date: Fri Feb 24 17:08:59 2017
ProcEnviron:
TERM=screen
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=C
SHELL=/bin/bash
SourcePackage: linux-hwe
UpgradeStatus: No upgrade log present (probably fresh install)

Tags:

Revision history for this message

Jeff Lane  (bladernr) wrote on 2017-02-24:

Dependencies.txt Edit (2.3 KiB, text/plain; charset="utf-8")
JournalErrors.txt Edit (114.3 KiB, text/plain; charset="utf-8")

Revision history for this message

Dimitri John Ledkov (xnox) wrote on 2017-02-27:

The hypervisor does not report to the linux kernel what the hypervisor network connections are configured with.

Please open an issue with IBM to improve z/VM hypervisor to report real line speeds to the guest linux.

Note, z/VM guests are effectively virtual machines, not physical hardware.

Changed in linux-hwe (Ubuntu):
status:	New → Won't Fix

Revision history for this message

Dimitri John Ledkov (xnox) wrote on 2017-02-27:

This is known issue/deficiency of the z/VM and the linux kernel.

Revision history for this message

Jeff Lane  (bladernr) wrote on 2017-02-27: Re: [Bug 1667828] Re: Network device on z/VM instance incorrectly reports line speed

Download full text (5.9 KiB)

So where does that 10Mb/s speed come from that ethtool is indicating?
If the hypervisor is providing no indication of what the device is
configured to provide, that's fine. BUT the kernel should not then be
simply making stuff up to fill in data.

For example, this is on zKVM:

Settings for eth0:
Supported ports: [ ]
Supported link modes: Not reported
Supported pause frame use: No
Supports auto-negotiation: No
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Speed: Unknown!
Duplex: Unknown! (255)
Port: Other
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Cannot get wake-on-lan settings: Operation not permitted
Link detected: yes

Just like on z/VM, the virtio device has no idea what it is capable of
nor what speed it is connected at. HOWEVER, it also doesn't lie about
what speed it's connected at, it simply says "Unknown!" which is
perfectly acceptable for these situations. This IS a valid kernel
bug, as ethtool (and sysfs) are presenting patently false data.

If the kernel doesn't know what speed it's connected at, that's
perfectly find, I can live with that and even work with it. But flat
out providing false data is wrong.

On Mon, Feb 27, 2017 at 6:54 AM, Dimitri John Ledkov
<email address hidden> wrote:
> This is known issue/deficiency of the z/VM and the linux kernel.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1667828
>
> Title:
> Network device on z/VM instance incorrectly reports line speed
>
> Status in linux-hwe package in Ubuntu:
> Won't Fix
>
> Bug description:
> Line speed is incorrectly reported on z/VM instances in /sys when
> compared to what is claimed to be supported and advertised.
>
> for example, ethtool shows supported and advertised speeds of 1000Mb/s
> but /sys is only indicating 10Mb/s.
>
> ubuntu@hwe0008:~$ ethtool enc600
> Settings for enc600:
> Supported ports: [ TP ]
> Supported link modes: 10baseT/Half 10baseT/Full
> 100baseT/Half 100baseT/Full
> 1000baseT/Half 1000baseT/Full
> Supported pause frame use: No
> Supports auto-negotiation: Yes
> Advertised link modes: 10baseT/Half 10baseT/Full
> 100baseT/Half 100baseT/Full
> 1000baseT/Half 1000baseT/Full
> Advertised pause frame use: No
> Advertised auto-negotiation: Yes
> Speed: 10Mb/s
> Duplex: Full
> Port: Twisted Pair
> PHYAD: 0
> Transceiver: internal
> Auto-negotiation: on
> MDI-X: Unknown
> Cannot get wake-on-lan settings: Operation not permitted
> Link detected: yes
> ubuntu@hwe0008:~$ cat /sys/class/net/enc600/speed
> 10
>
> I'm presuming this issue lies somewhere in the qeth driver:
> P: /devices/qeth/0.0.0600/net/enc600
> E: DEVPATH=/devices/qeth/0.0.0600/net/enc600
> E: ID_NET_DRIVER=qeth_l2
> E: ID_NET_LINK_FILE=/lib/systemd/network/99-default.link
> E: ID_NET_NAME=enc600
> E: ID_NET_NAME_MA...

So where does that 10Mb/s speed come from that ethtool is indicating?
If the hypervisor is providing no indication of what the device is
configured to provide, that's fine.  BUT the kernel should not then be
simply making stuff up to fill in data.

For example, this is on zKVM:

Settings for eth0:
Supported ports: [ ]
Supported link modes:   Not reported
Supported pause frame use: No
Supports auto-negotiation: No
Advertised link modes:  Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Speed: Unknown!
Duplex: Unknown! (255)
Port: Other
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Cannot get wake-on-lan settings: Operation not permitted
Link detected: yes

Just like on z/VM, the virtio device has no idea what it is capable of
nor what speed it is connected at.  HOWEVER, it also doesn't lie about
what speed it's connected at, it simply says "Unknown!" which is
perfectly acceptable for these situations.  This IS a valid kernel
bug, as ethtool (and sysfs) are presenting patently false data.

If the kernel doesn't know what speed it's connected at, that's
perfectly find, I can live with that and even work with it.  But flat
out providing false data is wrong.

On Mon, Feb 27, 2017 at 6:54 AM, Dimitri John Ledkov
<launchpad@surgut.co.uk> wrote:
> This is known issue/deficiency of the z/VM and the linux kernel.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1667828
>
> Title:
>   Network device on z/VM instance incorrectly reports line speed
>
> Status in linux-hwe package in Ubuntu:
>   Won't Fix
>
> Bug description:
>   Line speed is incorrectly reported on z/VM instances in /sys when
>   compared to what is claimed to be supported and advertised.
>
>   for example, ethtool shows supported and advertised speeds of 1000Mb/s
>   but /sys is only indicating 10Mb/s.
>
>   ubuntu@hwe0008:~$ ethtool enc600
>   Settings for enc600:
>           Supported ports: [ TP ]
>           Supported link modes:   10baseT/Half 10baseT/Full
>                                   100baseT/Half 100baseT/Full
>                                   1000baseT/Half 1000baseT/Full
>           Supported pause frame use: No
>           Supports auto-negotiation: Yes
>           Advertised link modes:  10baseT/Half 10baseT/Full
>                                   100baseT/Half 100baseT/Full
>                                   1000baseT/Half 1000baseT/Full
>           Advertised pause frame use: No
>           Advertised auto-negotiation: Yes
>           Speed: 10Mb/s
>           Duplex: Full
>           Port: Twisted Pair
>           PHYAD: 0
>           Transceiver: internal
>           Auto-negotiation: on
>           MDI-X: Unknown
>   Cannot get wake-on-lan settings: Operation not permitted
>           Link detected: yes
>   ubuntu@hwe0008:~$ cat /sys/class/net/enc600/speed
>   10
>
>   I'm presuming this issue lies somewhere in the qeth driver:
>   P: /devices/qeth/0.0.0600/net/enc600
>   E: DEVPATH=/devices/qeth/0.0.0600/net/enc600
>   E: ID_NET_DRIVER=qeth_l2
>   E: ID_NET_LINK_FILE=/lib/systemd/network/99-default.link
>   E: ID_NET_NAME=enc600
>   E: ID_NET_NAME_MAC=enx02280b000004
>   E: ID_NET_NAME_PATH=enc600
>   E: ID_PATH=ccwgroup-0.0.0600
>   E: ID_PATH_TAG=ccwgroup-0_0_0600
>   E: IFINDEX=2
>   E: INTERFACE=enc600
>   E: SUBSYSTEM=net
>   E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/enc600 /sys/subsystem/net/devices/enc600
>   E: TAGS=:systemd:
>   E: USEC_INITIALIZED=3784115
>
>   This is not an issue on my LPAR that uses the same drivers (but also
>   adds ctcm and zfcp, so maybe therein lies the difference in speed
>   reporting?)
>
>   In any case, if the speeds cannot be correctly reported, they shouldn't be reported at all.  virtio handles this by setting speed to -1:
>   ubuntu@s1lp9g003:~$ cat /sys/class/net/eth0/speed
>   -1
>   ubuntu@s1lp9g003:~$ ethtool eth0
>   Settings for eth0:
>         Supported ports: [ ]
>         Supported link modes:   Not reported
>         Supported pause frame use: No
>         Supports auto-negotiation: No
>         Advertised link modes:  Not reported
>         Advertised pause frame use: No
>         Advertised auto-negotiation: No
>         Speed: Unknown!
>         Duplex: Unknown! (255)
>         Port: Other
>         PHYAD: 0
>         Transceiver: internal
>         Auto-negotiation: off
>   Cannot get wake-on-lan settings: Operation not permitted
>         Link detected: yes
>
>   That last example is a 1Gb/s interface on a z/KVM instance.
>
>   ProblemType: Bug
>   DistroRelease: Ubuntu 16.04
>   Package: linux-image-4.8.0-36-generic 4.8.0-36.36~16.04.1
>   ProcVersionSignature: Ubuntu 4.8.0-36.36~16.04.1-generic 4.8.11
>   Uname: Linux 4.8.0-36-generic s390x
>   ApportVersion: 2.20.1-0ubuntu2.5
>   Architecture: s390x
>   Date: Fri Feb 24 17:08:59 2017
>   ProcEnviron:
>    TERM=screen
>    PATH=(custom, no user)
>    XDG_RUNTIME_DIR=<set>
>    LANG=C
>    SHELL=/bin/bash
>   SourcePackage: linux-hwe
>   UpgradeStatus: No upgrade log present (probably fresh install)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux-hwe/+bug/1667828/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: distribution=ubuntu; sourcepackage=linux-hwe; component=main; status=Won't Fix; importance=Undecided; assignee=None;
> Launchpad-Bug-Tags: apport-bug s390x xenial
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: bladernr xnox
> Launchpad-Bug-Reporter: Jeff Lane (bladernr)
> Launchpad-Bug-Modifier: Dimitri John Ledkov (xnox)
> Launchpad-Message-Rationale: Subscriber
> Launchpad-Message-For: bladernr

-- 
"Entropy isn't what it used to be."

Jeff Lane -
Server Certification Lead, Warrior Poet, Biker, Lover of Pie
Phone: 919-442-8649
Ubuntu Ham: W4KDH                          Freenode IRC: bladernr or bladernr_
gpg: 1024D/3A14B2DD 8C88 B076 0DD7 B404 1417  C466 4ABD 3635 3A14 B2DD

Changed in linux-hwe (Ubuntu):
status:	Won't Fix → Opinion

Revision history for this message

Jeff Lane  (bladernr) wrote on 2017-02-27:

I'll set this to Opinon for now, but it NEEDS to be take seriously. The kernel should not be lying about things like this. The speed should be either accurate (difficult given the explanation you've laid out) or it should be plainly marked as "Unknown" or "Unavilable". Just making up a valid number is NOT the right answer here.

Revision history for this message

Dimitri John Ledkov (xnox) wrote on 2017-02-27:

Frank could you please reverse proxy this bug back to IBM engineering, to make kernel report "Unknown" line speeds for qeth devices under Z/VM?

For reference, there are also confusion/requests w.r.t. this on linux-s390 maryst mailing list.

Regards,

Dimitri.

affects:	linux-hwe (Ubuntu) → linux (Ubuntu)
Changed in linux (Ubuntu):
status:	Opinion → Confirmed

Frank Heimes (fheimes) on 2017-02-27

Changed in ubuntu-z-systems:
assignee:	nobody → bugproxy (bugproxy)

bugproxy (bugproxy) on 2017-03-01

tags:

added: architecture-s39064 bugnameltc-152102 severity-high targetmilestone-inin1604

Frank Heimes (fheimes) on 2017-03-01

Changed in ubuntu-z-systems:
status:	New → Confirmed

Revision history for this message

bugproxy (bugproxy) wrote on 2017-03-02: Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2017-03-02 11:01 EDT-------
[CAN] Please check if z/VM APAR VM65785 / PTF UM34782 is installed on your machine....

Revision history for this message

bugproxy (bugproxy) wrote on 2017-03-06:

------- Comment From <email address hidden> 2017-03-06 05:03 EDT-------
Checking the zVM APAR is important, if the qeth-device is a virtual NIC in zVM. Is this the case? Could you please provide "lsqeth enc600" results?

Revision history for this message

Dimitri John Ledkov (xnox) wrote on 2017-03-06:

ubuntu@DEVAC02:~$ ethtool enc600
Settings for enc600:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
                         100baseT/Half 100baseT/Full
                         1000baseT/Half 1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
                         100baseT/Half 100baseT/Full
                         1000baseT/Half 1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 10Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown
Cannot get wake-on-lan settings: Operation not permitted
Link detected: yes
ubuntu@DEVAC02:~$ lsqeth enc600
Device name : enc600
-------------------------------------------------------------------------
        card_type : Virt.NIC QDIO
        cdev0 : 0.0.0600
        cdev1 : 0.0.0601
        cdev2 : 0.0.0602
        chpid : 00
        online : 1
        portname : no portname required
        portno : 0
        state : UP (LAN ONLINE)
        priority_queueing : always queue 2
        buffer_count : 64
        layer2 : 1
        isolation : none
        bridge_role : none
        bridge_state : inactive
        bridge_hostnotify : 0
        bridge_reflect_promisc : none

I'm not sure how to check zVM APAR, I've requested assistance for that.

Revision history for this message

Dimitri John Ledkov (xnox) wrote on 2017-03-06:

#10

We currently believe our firmware is not up to date, as per comments above. We will investigate our installs further, upgrade if necessory, and retest. Setting to incomplete, and assigning to frank heimes to coordinate this effort.

Changed in ubuntu-z-systems:
assignee:	bugproxy (bugproxy) → Frank Heimes (frank-heimes)
Changed in linux (Ubuntu):
status:	Confirmed → Incomplete
Changed in ubuntu-z-systems:
status:	Confirmed → Incomplete

Revision history for this message

Frank Heimes (fheimes) wrote on 2017-03-07:

#11

We currently don't have APAR VM65785 or PTF UM34782 installed in our z/VM 6.3, because it looks like it's more z/OS and IP v6 related:
- according to the brief description "VSwitch error x'E00A' for IPv6 Address In use on LAN" (taken from here: http://www.vm.ibm.com/virtualnetwork/mntlvl63.html
- and also from the detailed description: http://www-01.ibm.com/support/docview.wss?uid=isg1VM65785

Anyway, if this APAR goes beyond the description we may consider to apply this.

Revision history for this message

bugproxy (bugproxy) wrote on 2017-03-07:

#12

------- Comment From <email address hidden> 2017-03-07 03:38 EDT-------
Frank, please note the second part of the APAR description:

2: When a virtual NIC reflects the characteristics of the
NIC via the QUERY_CARD_INFO assist, the port speed and lan
type could be improved.

Problem conclusion
2: The QUERY_CARD_INFO simulation was updated to reflect a
LAN type of Fiber and a speed of 10 Gigabit.

Revision history for this message

Frank Heimes (fheimes) wrote on 2017-03-09:

#13

Looks like the problem is indeed fixed by APAR VM65785:

$ ethtool enc600
Settings for enc600:
Supported ports: [ FIBRE ]
Supported link modes: 10baseT/Half 10baseT/Full
                         100baseT/Half 100baseT/Full
                         1000baseT/Half 1000baseT/Full
                         10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
                         100baseT/Half 100baseT/Full
                         1000baseT/Half 1000baseT/Full
                         10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Cannot get wake-on-lan settings: Operation not permitted
Link detected: yes

$ cat /sys/class/net/enc600/speed
10000
$

Frank Heimes (fheimes) on 2017-03-09

Changed in ubuntu-z-systems:
status:	Incomplete → Fix Released
Changed in linux (Ubuntu):
status:	Incomplete → Fix Released

Revision history for this message

bugproxy (bugproxy) wrote on 2017-03-09:

#14

------- Comment From <email address hidden> 2017-03-09 11:23 EDT-------
After installation of the z/VM APAR the problem was solved.
IBM Bugzilla-Status -> closed.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntulinux package

Network device on z/VM instance incorrectly reports line speed

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
linux package