ubuntu-installer makes a contradictional statement regarding layer2/layer3

[bugproxy]

== Comment: #0 - Thorsten Diehl <email address hidden> - 2016-02-03 12:05:19 ==
When I configure a qeth OSA device, the installer asks for device type, device and protocol layer:

Please choose the type of your primary network interface that you will need for

installing the Debian system (via NFS or HTTP). Only the listed devices are
supported.
Network device type:
  1: ctc: Channel to Channel (CTC) or ESCON connection,
  2: qeth: OSA-Express in QDIO mode / HiperSockets,
  3: iucv: Inter-User Communication Vehicle - available for VM guests only,

  4: virtio: KVM VirtIO,
Prompt: '?' for help> 2
2

Please select the OSA-Express QDIO / HiperSockets device.
Device:
  1: 0.0.f5f0-0.0.f5f1-0.0.f5f2 [*],
Prompt: '?' for help, default=1> 1
1

By default OSA-Express cards use layer3 mode. In that mode LLC headers are
removed from incoming IPv4 packets. Using the card in layer2 mode will make it
keep the MAC addresses of IPv4 packets.
Use this device in layer2 mode?
  1: Yes [*] 2: No
Prompt: '?' for help, default=1>

The last question experienced a change according to https://bugs.launchpad.net/ubuntu/+source/s390-netdevice/+bug/1526801
Layer2 is the default, which is correct, since the qeth driver uses layer2 as default as well.

However, the introductional text says: "By default OSA-Express cards use layer3 mode."

And this is wrong and needs to be fixed. Correct wording is:

"By default OSA-Express cards use layer2 mode. In layer3 mode LLC headers are
removed from incoming IPv4 packets. Using the card in layer2 mode will make it
keep the MAC addresses of IPv4 packets."

== Comment: #2 - Thorsten Diehl <email address hidden> - 2016-02-03 12:08:52 ==
Although LP 1526801 deals with this subject, it is different.
LP 1526801 is implemented and closed. Now the documentational part in the installer needs to follow that. Thus I recommend to treat this as a separate bug.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-02-04 02:52 EDT-------
HI,

(In reply to comment #0)
> When I configure a qeth OSA device, the installer asks for device type,
> device and protocol layer:
>
> Please choose the type of your primary network interface that you will need
> for
>
> installing the Debian system (via NFS or HTTP). Only the listed devices are
>
> supported.
> Network device type:
> 1: ctc: Channel to Channel (CTC) or ESCON connection,
> 2: qeth: OSA-Express in QDIO mode / HiperSockets,
> 3: iucv: Inter-User Communication Vehicle - available for VM guests only,
>
>
> 4: virtio: KVM VirtIO,
> Prompt: '?' for help> 2
> 2
>
> Please select the OSA-Express QDIO / HiperSockets device.
> Device:
> 1: 0.0.f5f0-0.0.f5f1-0.0.f5f2 [*],
> Prompt: '?' for help, default=1> 1
> 1
>
> By default OSA-Express cards use layer3 mode. In that mode LLC headers are
> removed from incoming IPv4 packets. Using the card in layer2 mode will make
> it
> keep the MAC addresses of IPv4 packets.
> Use this device in layer2 mode?
> 1: Yes [*] 2: No
> Prompt: '?' for help, default=1>
>
> The last question experienced a change according to
> https://bugs.launchpad.net/ubuntu/+source/s390-netdevice/+bug/1526801
> Layer2 is the default, which is correct, since the qeth driver uses layer2
> as default as well.
>

I wonder why the default has been changed at all. The default is just a guess. The s390-netdevice does not have any heuristics to determine a potential correct default and I would have to investigate if that is possible.

The system administrator should provide information about whether layer2 or layer2 is used for a particular network interface. Statement like "By default OSA-Express cards" are incorrect as the modes can be configured. The same applies for HiperSocket devices. And also there is real and virtual OSA/HiperSockets.

One last note, if you misconfigure the layer2 setting you want get a network connection.

[Dimitri John Ledkov (xnox)]

Hello,

Ideally we would use heuristics. For example, does the hardware and the kernel know about layer2 already and e.g. does one need to change /sys/devices/qeth/0.0.0600/layer2 at all? Ideally layer2 would be set to the correct value by the kernel, and we simply lower the priority of this question, and thus stop asking users about it by default. I'm now digging into the kernel code and it looks like qeth_core_main loads layer2 by default (unless I'm missing something, I'm not a kernel developer).

I'm happy to adjust statement to a more technically correct one.
"""
Use this device in layer2 mode?
The qeth network driver is used for OSA-Express cards, HiperSockets and similar devices. Using the card in layer2 mode will make it keep the MAC addresses of IPv4 packets. Alternative is to use layer3 mode, in that mode LLC headers are removed from incoming IPv4 packets. Incorrect mode will lead to lack of connectivity.
"""
Comments on the above?

On the desktop installer, we check the connectivity by retrieving and validating check-sum of http://start.ubuntu.com/connectivity-check and thus really validate that one can access things (but i guess most mainframes are fire-walled to prevent such access). I guess in d-i context it's about being able to access the network mirror to continue installation. Not sure how we can validate automatically, given that local mirror information may not be available yet.

Lastly, I've changed the default based on the statistically insignificant survey of three people ("most mainframes use layer2, something like 60-80% probably). And a personal feeling that a bunch of things require layer2 network. The fact that our mainframe is configured with layer2 network cards was a factor too =) to ease things for us. Defaults should be sensible, or what we want the future be like.

1) ideally kernel and/or with udev assistance would do the autodiscovery of the required mode, if possible.
2) does the proposed new wording sound better to you? or how would you like the question to be worded?
3) should I keep layer2 as default?

Regards,

Dimitri.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-02-05 10:37 EDT-------
Hi Dimitri,

well done, your investigation and research! Thank you that you took the time to dig deeply into that, even into kernel code. Great work!

Here my answers:

IIRC the kernel has for the qeth module layer2=1 set as the default. To use layer3 (layer2=0), it needs to be set explicitely. Which means, if noone changes the sysfs entry for layer2, layer 2 will be used as default. For some setups it might be required to have layer3, thus it is mandatory to ask.

An configuration check like for x64 arch will usually not work in a mainframe/enterprise environment, may it be due to the lack of dhcp server or to stringent firewalls.

And now to your last three questions:

Q1: ideally kernel and/or with udev assistance would do the autodiscovery of the required mode, if possible.
A1: yes, that would be preferable, but IIRC for qeth is is the user's responsibilty to choose the correct layer2 setting, according to the network admin's advice.

> Use this device in layer2 mode?
> The qeth network driver is used for OSA-Express cards, HiperSockets and similar devices.
> Using the card in layer2 mode will make it keep the MAC addresses of IPv4 packets.
> Alternative is to use layer3 mode, in that mode LLC headers are removed from incoming
> IPv4 packets. Incorrect mode will lead to lack of connectivity.
Q2: does the proposed new wording sound better to you? or how would you like the question to be worded?
A2: Yes, good and crispy wording! I totally agree, but please move the question "Use this device in layer2 mode?" to the end, as it was before => and then it looks like this:
The qeth network driver is used for OSA-Express cards, HiperSockets and similar devices. Using the card in
layer2 mode will make it keep the MAC addresses of IPv4 packets. Alternative is to use layer3 mode, in that
mode LLC headers are removed from incoming IPv4 packets. Incorrect mode will lead to lack of connectivity.
Use this device in layer2 mode?

Q3: should I keep layer2 as default?
A3: yes, please. It is for almost all cases the recommended layer mode.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-02-05 11:40 EDT-------
We have to distinguish between the different device types supported by the qeth driver:
1. OSA-cards: the qeth default is layer2. The OSA-card is able to convert between layer2 and layer3. Thus no lack of connectivity here. Nevertheless a customer may want to select layer3, especially if he wants to make use of TCP Segmentation Offload.
2. HiperSockets: the qeth default is layer3 to allow smooth communication with zOS. HiperSockets LANs are either layer2 or layer3; there is no connectivity between a layer2 participant and a layer3 participant.
3.virtual NICs attached to a z/VM VSWITCH or GuestLAN when Linux is running as a z/VM guest: Here the Linux definition has to follow the layer definition of the VSWITCH or GuestLAN in z/VM, otherwise connectivity is broken. Up to now there is no automatic detection of the layer defined in z/VM.

[Dimitri John Ledkov (xnox)]

Is the card type at all exposed anywhere?

E.g:
cat /sys/bus/ccwgroup/devices/0.0.0600/card_type

For me says - Virt.NIC QDIO, which sounds like case 3 (this is z/VM). I don't think i have access to OSA-card direct, or Hipersockets.

I guess we could sense on card_type and use it in the question. However there are a few, aren't there looking at the source code there are a few:

https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/unstable/tree/drivers/s390/net/qeth_core_main.c#n137

Virt.NIC QDIO
Virt.NIC Hiper
Virt.NIC OSM
Virt.NIC OSX
unknown
OSD_100
HSTR
OSD_1000
OSD_10GIG
OSD_FE_LANE
OSD_TR_LANE
OSD_GbE_LANE
OSD_ATM_LANE
OSD_Express
HiperSockets
OSN
OSM_1000
OSX_10GIG

I guess layer2 is a good default for most of them. Which of the above do you believe layer3 is a better guess? And then working that into the question would be good:

(In d-i "question" comes first, then decription, then answers. This particular question is a boolean yes/no question -> true (l2) or false (l3), changing this will result in breaking compatibility with preseed files)

"""
-> Select layer for Virt.Nic Hiper card

The qeth network driver is about to be configured for the Virt.Nic Hiper card. The driver and the card can use either OSI model Layer 2 or Layer 3 modes. And this setting should match the network configuration this card is attached to. Using the card in Layer 2 mode will make it keep the MAC addresses of the IPv4 packets. Alternative is to use Layer 3 mode, in that mode LLC headers are removed from the incoming IPv4 packets. Incorrect mode may lead to lack of connectivity. Use this card in Layer 2 mode?

Default [Y]: Yes / No
""

The Virt.Nic Hiper card is just an example, as actually variable will be replaced with the contents of card_type sysfs property, which should be available at that point. The full list of possible card types, from current kernel sources, is shown above.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-02-08 05:39 EDT-------
Virt.NIC QDIO - depends on the z/VM layer configured, more details below
Virt.NIC Hiper - always layer3
Virt.NIC OSM - that's dead code, does not occur, I will add the removal on my to do list, any default is ok
Virt.NIC OSX - that's dead code, does not occur, I will add the removal on my to do list, any default is ok
unknown - should hopefully not occur, thus any default is ok
OSD_100 - both layers possible, use layer2 as default
HSTR - token ring no longer supported, does not occur, has been layer3 only
OSD_1000 - both layers possible, use layer2 as default
OSD_10GIG - both layers possible, use layer2 as default
OSD_FE_LANE - outdated type, does no longer occur, has been layer3 only
OSD_TR_LANE - token ring no longer supported, does not occur, has been layer3 only
OSD_GbE_LANE - outdated type, does no longer occur, has been layer3 only
OSD_ATM_LANE - outdated type, does no longer occur, has been layer3 only
OSD_Express - both layers possible, use layer2 as default
HiperSockets - both layers possible, use layer3 as default
OSN - special case, needed for an IBM product already approaching end of life time (end of service effective 2016-03-31), does not have a layer2 attribute at all, ignore?
OSM_1000 - always layer2
OSX_10GIG - both layers possible, use layer2 as default

As you see, there are still some outdated OSA-types listed, which we have not yet removed to still enable running on old hardware. They are not critical, and you can configure them as you like.

A problematic type is "Virt.NIC QDIO". Here qeth does not yet determine the layer of the z/VM VSWITCH automatically. zVM commands allow to determine the layer. You could first determine the name of the VSWITCH, a virtual NIC is connected to - with "vmcp q virtual nic <vdev>" .
Sample:
> vmcp QUERY VIRTUAL NIC f5f0
Adapter F5F0.P00 Type: QDIO Name: HYD1G1 Devices: 3
MAC: 02-12-34-56-78-90 VSWITCH: SYSTEM VSW444
Here "VSW444" is the name of the VSWITCH. Then you can determine the layer of the VSWITCH with "vmcp q vswitch <vswitch name>".
Sample:
> vmcp QUERY VSWITCH VSW444
VSWITCH SYSTEM VSW444 Type: QDIO Connected: 9 Maxconn: INFINITE
PERSISTENT RESTRICTED ETHERNET Accounting: OFF
...
The keyword here is "ETHERNET". In this case layer2 must be configured for the virtual NIC, otherwise layer3.

[Dimitri John Ledkov (xnox)]

Thank you for all of these details. I believe there is sufficient information to improve network device configuration. Marking bug as trianged, and setting appropriate priority for this usability issue.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-02-08 07:33 EDT-------
Hi,

thanks for this important information. It would advice to list these also in the Device Drivers Book. I have seen that information not before and it could help to improve the implementation of the s390-netdevice.

(In reply to comment #10)
> Virt.NIC QDIO - depends on the z/VM layer configured, more details below
> Virt.NIC Hiper - always layer3
> Virt.NIC OSM - that's dead code, does not occur, I will add the removal on
> my to do list, any default is ok
> Virt.NIC OSX - that's dead code, does not occur, I will add the removal on
> my to do list, any default is ok
> unknown - should hopefully not occur, thus any default is ok
> OSD_100 - both layers possible, use layer2 as default
> HSTR - token ring no longer supported, does not occur, has been layer3 only
> OSD_1000 - both layers possible, use layer2 as default
> OSD_10GIG - both layers possible, use layer2 as default
> OSD_FE_LANE - outdated type, does no longer occur, has been layer3 only
> OSD_TR_LANE - token ring no longer supported, does not occur, has been
> layer3 only
> OSD_GbE_LANE - outdated type, does no longer occur, has been layer3 only
> OSD_ATM_LANE - outdated type, does no longer occur, has been layer3 only
> OSD_Express - both layers possible, use layer2 as default
> HiperSockets - both layers possible, use layer3 as default
> OSN - special case, needed for an IBM product already approaching end of
> life time (end of service effective 2016-03-31), does not have a layer2
> attribute at all, ignore?
> OSM_1000 - always layer2
> OSX_10GIG - both layers possible, use layer2 as default
>
> As you see, there are still some outdated OSA-types listed, which we have
> not yet removed to still enable running on old hardware. They are not
> critical, and you can configure them as you like.

Is there a sysfs attribute to get a numeric type for these or does an implementation needs to parse the strings?

>
> A problematic type is "Virt.NIC QDIO". Here qeth does not yet determine the
> layer of the z/VM VSWITCH automatically. zVM commands allow to determine the
> layer. You could first determine the name of the VSWITCH, a virtual NIC is
> connected to - with "vmcp q virtual nic <vdev>" .
> Sample:
> > vmcp QUERY VIRTUAL NIC f5f0
> Adapter F5F0.P00 Type: QDIO Name: HYD1G1 Devices: 3
> MAC: 02-12-34-56-78-90 VSWITCH: SYSTEM VSW444
> Here "VSW444" is the name of the VSWITCH. Then you can determine the layer
> of the VSWITCH with "vmcp q vswitch <vswitch name>".
> Sample:
> > vmcp QUERY VSWITCH VSW444
> VSWITCH SYSTEM VSW444 Type: QDIO Connected: 9 Maxconn: INFINITE
> PERSISTENT RESTRICTED ETHERNET Accounting: OFF
> ...
> The keyword here is "ETHERNET". In this case layer2 must be configured for
> the virtual NIC, otherwise layer3.

AFAIK, the CP QUERY VSWITCH is a privileged command and, thus, an implementation should not rely on getting this information.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-02-08 09:34 EDT-------
> I would advice to list these also in the Device Drivers Book.
Part of my given information is already contained in the qeth chapter of the Device Drivers Book under "Finding out the type of a network adapter" and under "Setting the layer2 attribute".

> Is there a sysfs attribute to get a numeric type for these or does an implementation needs to parse the strings?
No, there is just the sysfs attribute "card_type" with its string-type output.

> AFAIK, the CP QUERY VSWITCH is a privileged command and, thus, an implementation should not rely on getting this information.
According to zVM documentation it should work here: "A class G user can view information about a virtual switch if the user is in the CP access list for the virtual switch, or if the user currently has an adapter connected to it."

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-03-09 06:07 EDT-------
Dimitri, after some time we decided, that for now the actually implemented wording is fine, sufficient and fixes the original issue.

------- Comment From <email address hidden> 2016-03-09 06:08 EDT-------
Successfully verified (with a long delay) on ubuntu installer version 431. Closed.

[Dimitri John Ledkov (xnox)]

As per above comment downgrading to wishlist, imho there are still useful things that could be implemented from the above discussion, if not as urgent as originally was deemed.

[Dimitri John Ledkov (xnox)]

closing on LP too, as is closed on IBM side.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-01-10 07:03 EDT-------
IBM Bugzilla -> closed