Ubuntu
systemd package

[Ubuntu 20.4.2] vLan not getting static IP assigned (on s390x)

Bug #1929657 reported by bugproxy
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Fix Released
High
Skipper Bug Screeners
linux (Ubuntu)
Invalid
Undecided
Unassigned
netplan.io (Ubuntu)
Invalid
Undecided
Unassigned
systemd (Ubuntu)
Invalid
Undecided
Canonical Foundations Team

Bug Description

---Problem Description---
Doing vLAN configuration one of the vLANs is not getting static IP assigned when the rest are workin without problems

Contact Information = Mario <email address hidden>

---uname output---
Linux ilabg13.tuc.stglabs.ibm.com 5.4.0-73-generic #82-Ubuntu SMP Wed Apr 14 17:29:32 UTC 2021 s390x s390x s390x GNU/Linux

Machine Type = z15

---Debugger---
A debugger is not configured

---Steps to Reproduce---
 1)Configure the netplan file as follow:
root@ilabg13:~# cat /etc/netplan/01-iscsi-config.yaml

# This is the network config written by 'subiquity'

network:

  ethernets:

    encdb0:

      addresses:

      - 11.111.114.213/22

      macaddress: 02:76:54:00:00:03

    encdc0:

      addresses:

      - 11.111.112.213/22

      macaddress: 02:76:54:00:00:04

    enP50s3832 :

      addresses:

      - 11.111.112.214/22

    enP53p0s0:

      addresses:

      - 11.111.112.215/22

  vlans:

    encdb0.160:

      id: 160

      link: encdb0

      mtu: 9000

      addresses:

      - 192.168.160.53/24

    encdc0.150:

      id: 150

      link: encdc0

      mtu: 9000

      addresses:

      - 192.168.150.53/24

    enP50s3832.170:

      id: 170

      link: enP50s3832

      mtu: 9000

      addresses:

      - 192.168.170.53/24

    enP53p0s0.171:

      id: 171

      link: enP53p0s0

      mtu: 9000

      addresses:

      - 192.168.171.53/24

  version: 2

2)run net plan apply:
root@ilabg13:~# netplan --debug apply

** (generate:59965): DEBUG: 14:55:15.046: Processing input file /etc/netplan/00-installer-config.yaml..

** (generate:59965): DEBUG: 14:55:15.046: starting new processing pass

** (generate:59965): DEBUG: 14:55:15.046: Processing input file /etc/netplan/01-iscsi-config.yaml..

** (generate:59965): DEBUG: 14:55:15.046: starting new processing pass

** (generate:59965): DEBUG: 14:55:15.046: We have some netdefs, pass them through a final round of validation

** (generate:59965): DEBUG: 14:55:15.046: encdc0: setting default backend to 1

** (generate:59965): DEBUG: 14:55:15.046: Configuration is valid

** (generate:59965): DEBUG: 14:55:15.046: encdb0: setting default backend to 1

** (generate:59965): DEBUG: 14:55:15.046: Configuration is valid

** (generate:59965): DEBUG: 14:55:15.046: ence0f: setting default backend to 1

** (generate:59965): DEBUG: 14:55:15.046: Configuration is valid

** (generate:59965): DEBUG: 14:55:15.046: encdb0.160: setting default backend to 1

** (generate:59965): DEBUG: 14:55:15.046: Configuration is valid

** (generate:59965): DEBUG: 14:55:15.046: enP50s3832: setting default backend to 1

** (generate:59965): DEBUG: 14:55:15.046: Configuration is valid

** (generate:59965): DEBUG: 14:55:15.046: enP53p0s0.171: setting default backend to 1

** (generate:59965): DEBUG: 14:55:15.046: Configuration is valid

** (generate:59965): DEBUG: 14:55:15.046: enP50s3832.170: setting default backend to 1

** (generate:59965): DEBUG: 14:55:15.046: Configuration is valid

** (generate:59965): DEBUG: 14:55:15.046: enP53p0s0: setting default backend to 1

** (generate:59965): DEBUG: 14:55:15.046: Configuration is valid

** (generate:59965): DEBUG: 14:55:15.046: encdc0.150: setting default backend to 1

** (generate:59965): DEBUG: 14:55:15.046: Configuration is valid

** (generate:59965): DEBUG: 14:55:15.047: Generating output files..

** (generate:59965): DEBUG: 14:55:15.047: openvswitch: definition ence0f is not for us (backend 1)

** (generate:59965): DEBUG: 14:55:15.047: NetworkManager: definition ence0f is not for us (backend 1)

** (generate:59965): DEBUG: 14:55:15.047: openvswitch: definition encdb0 is not for us (backend 1)

** (generate:59965): DEBUG: 14:55:15.047: NetworkManager: definition encdb0 is not for us (backend 1)

** (generate:59965): DEBUG: 14:55:15.047: openvswitch: definition encdc0 is not for us (backend 1)

** (generate:59965): DEBUG: 14:55:15.047: NetworkManager: definition encdc0 is not for us (backend 1)

** (generate:59965): DEBUG: 14:55:15.047: openvswitch: definition enP50s3832 is not for us (backend 1)

** (generate:59965): DEBUG: 14:55:15.047: NetworkManager: definition enP50s3832 is not for us (backend 1)

** (generate:59965): DEBUG: 14:55:15.047: openvswitch: definition enP53p0s0 is not for us (backend 1)

** (generate:59965): DEBUG: 14:55:15.047: NetworkManager: definition enP53p0s0 is not for us (backend 1)

** (generate:59965): DEBUG: 14:55:15.047: openvswitch: definition encdb0.160 is not for us (backend 1)

** (generate:59965): DEBUG: 14:55:15.047: NetworkManager: definition encdb0.160 is not for us (backend 1)

** (generate:59965): DEBUG: 14:55:15.047: openvswitch: definition encdc0.150 is not for us (backend 1)

** (generate:59965): DEBUG: 14:55:15.047: NetworkManager: definition encdc0.150 is not for us (backend 1)

** (generate:59965): DEBUG: 14:55:15.047: openvswitch: definition enP50s3832.170 is not for us (backend 1)

** (generate:59965): DEBUG: 14:55:15.047: NetworkManager: definition enP50s3832.170 is not for us (backend 1)

** (generate:59965): DEBUG: 14:55:15.047: openvswitch: definition enP53p0s0.171 is not for us (backend 1)

** (generate:59965): DEBUG: 14:55:15.047: NetworkManager: definition enP53p0s0.171 is not for us (backend 1)

(generate:59965): GLib-DEBUG: 14:55:15.047: posix_spawn avoided (fd close requested)

(generate:59965): GLib-DEBUG: 14:55:15.048: posix_spawn avoided (fd close requested)

DEBUG:netplan generated networkd configuration changed, restarting networkd

DEBUG:ence0f not found in {}

DEBUG:encdb0 not found in {'ence0f': {'addresses': ['9.11.116.213/24'], 'gateway4': '9.11.116.1', 'nameservers': {'addresses': ['9.11.227.25']}}}

DEBUG:encdc0 not found in {'ence0f': {'addresses': ['9.11.116.213/24'], 'gateway4': '9.11.116.1', 'nameservers': {'addresses': ['9.11.227.25']}}, 'encdb0': {'addresses': ['11.111.114.213/22'], 'macaddress': '02:76:54:00:00:03'}}

DEBUG:enP50s3832 not found in {'ence0f': {'addresses': ['9.11.116.213/24'], 'gateway4': '9.11.116.1', 'nameservers': {'addresses': ['9.11.227.25']}}, 'encdb0': {'addresses': ['11.111.114.213/22'], 'macaddress': '02:76:54:00:00:03'}, 'encdc0': {'addresses': ['11.111.112.213/22'], 'macaddress': '02:76:54:00:00:04'}}

DEBUG:enP53p0s0 not found in {'ence0f': {'addresses': ['9.11.116.213/24'], 'gateway4': '9.11.116.1', 'nameservers': {'addresses': ['9.11.227.25']}}, 'encdb0': {'addresses': ['11.111.114.213/22'], 'macaddress': '02:76:54:00:00:03'}, 'encdc0': {'addresses': ['11.111.112.213/22'], 'macaddress': '02:76:54:00:00:04'}, 'enP50s3832': {'addresses': ['11.111.112.214/22']}}

DEBUG:encdb0.160 not found in {}

DEBUG:encdc0.150 not found in {'encdb0.160': {'id': 160, 'link': 'encdb0', 'mtu': 9000, 'addresses': ['192.168.160.53/24']}}

DEBUG:enP50s3832.170 not found in {'encdb0.160': {'id': 160, 'link': 'encdb0', 'mtu': 9000, 'addresses': ['192.168.160.53/24']}, 'encdc0.150': {'id': 150, 'link': 'encdc0', 'mtu': 9000, 'addresses': ['192.168.150.53/24']}}

DEBUG:enP53p0s0.171 not found in {'encdb0.160': {'id': 160, 'link': 'encdb0', 'mtu': 9000, 'addresses': ['192.168.160.53/24']}, 'encdc0.150': {'id': 150, 'link': 'encdc0', 'mtu': 9000, 'addresses': ['192.168.150.53/24']}, 'enP50s3832.170': {'id': 170, 'link': 'enP50s3832', 'mtu': 9000, 'addresses': ['192.168.170.53/24']}}

DEBUG:Merged config:

network:

  ethernets:

    enP50s3832:

      addresses:

      - 11.111.112.214/22

    enP53p0s0:

      addresses:

      - 11.111.112.215/22

    encdb0:

      addresses:

      - 11.111.114.213/22

      macaddress: 02:76:54:00:00:03

    encdc0:

      addresses:

      - 11.111.112.213/22

      macaddress: 02:76:54:00:00:04

    ence0f:

      addresses:

      - 9.11.116.213/24

      gateway4: 9.11.116.1

      nameservers:

        addresses:

        - 9.11.227.25

  version: 2

  vlans:

    enP50s3832.170:

      addresses:

      - 192.168.170.53/24

      id: 170

      link: enP50s3832

      mtu: 9000

    enP53p0s0.171:

      addresses:

      - 192.168.171.53/24

      id: 171

      link: enP53p0s0

      mtu: 9000

    encdb0.160:

      addresses:

      - 192.168.160.53/24

      id: 160

      link: encdb0

      mtu: 9000

    encdc0.150:

      addresses:

      - 192.168.150.53/24

      id: 150

      link: encdc0

      mtu: 9000

DEBUG:no netplan generated NM configuration exists

DEBUG:ence0f not found in {}

DEBUG:encdb0 not found in {'ence0f': {'addresses': ['9.11.116.213/24'], 'gateway4': '9.11.116.1', 'nameservers': {'addresses': ['9.11.227.25']}}}

DEBUG:encdc0 not found in {'ence0f': {'addresses': ['9.11.116.213/24'], 'gateway4': '9.11.116.1', 'nameservers': {'addresses': ['9.11.227.25']}}, 'encdb0': {'addresses': ['11.111.114.213/22'], 'macaddress': '02:76:54:00:00:03'}}

DEBUG:enP50s3832 not found in {'ence0f': {'addresses': ['9.11.116.213/24'], 'gateway4': '9.11.116.1', 'nameservers': {'addresses': ['9.11.227.25']}}, 'encdb0': {'addresses': ['11.111.114.213/22'], 'macaddress': '02:76:54:00:00:03'}, 'encdc0': {'addresses': ['11.111.112.213/22'], 'macaddress': '02:76:54:00:00:04'}}

DEBUG:enP53p0s0 not found in {'ence0f': {'addresses': ['9.11.116.213/24'], 'gateway4': '9.11.116.1', 'nameservers': {'addresses': ['9.11.227.25']}}, 'encdb0': {'addresses': ['11.111.114.213/22'], 'macaddress': '02:76:54:00:00:03'}, 'encdc0': {'addresses': ['11.111.112.213/22'], 'macaddress': '02:76:54:00:00:04'}, 'enP50s3832': {'addresses': ['11.111.112.214/22']}}

DEBUG:encdb0.160 not found in {}

DEBUG:encdc0.150 not found in {'encdb0.160': {'id': 160, 'link': 'encdb0', 'mtu': 9000, 'addresses': ['192.168.160.53/24']}}

DEBUG:enP50s3832.170 not found in {'encdb0.160': {'id': 160, 'link': 'encdb0', 'mtu': 9000, 'addresses': ['192.168.160.53/24']}, 'encdc0.150': {'id': 150, 'link': 'encdc0', 'mtu': 9000, 'addresses': ['192.168.150.53/24']}}

DEBUG:enP53p0s0.171 not found in {'encdb0.160': {'id': 160, 'link': 'encdb0', 'mtu': 9000, 'addresses': ['192.168.160.53/24']}, 'encdc0.150': {'id': 150, 'link': 'encdc0', 'mtu': 9000, 'addresses': ['192.168.150.53/24']}, 'enP50s3832.170': {'id': 170, 'link': 'enP50s3832', 'mtu': 9000, 'addresses': ['192.168.170.53/24']}}

DEBUG:Merged config:

network:

  ethernets:

    enP50s3832:

      addresses:

      - 11.111.112.214/22

    enP53p0s0:

      addresses:

      - 11.111.112.215/22

    encdb0:

      addresses:

      - 11.111.114.213/22

      macaddress: 02:76:54:00:00:03

    encdc0:

      addresses:

      - 11.111.112.213/22

      macaddress: 02:76:54:00:00:04

    ence0f:

      addresses:

      - 9.11.116.213/24

      gateway4: 9.11.116.1

      nameservers:

        addresses:

        - 9.11.227.25

  version: 2

  vlans:

    enP50s3832.170:

      addresses:

      - 192.168.170.53/24

      id: 170

      link: enP50s3832

      mtu: 9000

    enP53p0s0.171:

      addresses:

      - 192.168.171.53/24

      id: 171

      link: enP53p0s0

      mtu: 9000

    encdb0.160:

      addresses:

      - 192.168.160.53/24

      id: 160

      link: encdb0

      mtu: 9000

    encdc0.150:

      addresses:

      - 192.168.150.53/24

      id: 150

      link: encdc0

      mtu: 9000

DEBUG:Link changes: {}

DEBUG:netplan triggering .link rules for lo

DEBUG:netplan triggering .link rules for encdb0

DEBUG:netplan triggering .link rules for ence0f

DEBUG:netplan triggering .link rules for encdc0

DEBUG:netplan triggering .link rules for enP50s3832

DEBUG:netplan triggering .link rules for enP53p0s0

DEBUG:netplan triggering .link rules for enP53p0s0.171

DEBUG:netplan triggering .link rules for enP50s3832.170

DEBUG:netplan triggering .link rules for encdc0.150

DEBUG:netplan triggering .link rules for encdb0.160

DEBUG:ence0f not found in {}

DEBUG:encdb0 not found in {'ence0f': {'addresses': ['9.11.116.213/24'], 'gateway4': '9.11.116.1', 'nameservers': {'addresses': ['9.11.227.25']}}}

DEBUG:encdc0 not found in {'ence0f': {'addresses': ['9.11.116.213/24'], 'gateway4': '9.11.116.1', 'nameservers': {'addresses': ['9.11.227.25']}}, 'encdb0': {'addresses': ['11.111.114.213/22'], 'macaddress': '02:76:54:00:00:03'}}

DEBUG:enP50s3832 not found in {'ence0f': {'addresses': ['9.11.116.213/24'], 'gateway4': '9.11.116.1', 'nameservers': {'addresses': ['9.11.227.25']}}, 'encdb0': {'addresses': ['11.111.114.213/22'], 'macaddress': '02:76:54:00:00:03'}, 'encdc0': {'addresses': ['11.111.112.213/22'], 'macaddress': '02:76:54:00:00:04'}}

DEBUG:enP53p0s0 not found in {'ence0f': {'addresses': ['9.11.116.213/24'], 'gateway4': '9.11.116.1', 'nameservers': {'addresses': ['9.11.227.25']}}, 'encdb0': {'addresses': ['11.111.114.213/22'], 'macaddress': '02:76:54:00:00:03'}, 'encdc0': {'addresses': ['11.111.112.213/22'], 'macaddress': '02:76:54:00:00:04'}, 'enP50s3832': {'addresses': ['11.111.112.214/22']}}

DEBUG:encdb0.160 not found in {}

DEBUG:encdc0.150 not found in {'encdb0.160': {'id': 160, 'link': 'encdb0', 'mtu': 9000, 'addresses': ['192.168.160.53/24']}}

DEBUG:enP50s3832.170 not found in {'encdb0.160': {'id': 160, 'link': 'encdb0', 'mtu': 9000, 'addresses': ['192.168.160.53/24']}, 'encdc0.150': {'id': 150, 'link': 'encdc0', 'mtu': 9000, 'addresses': ['192.168.150.53/24']}}

DEBUG:enP53p0s0.171 not found in {'encdb0.160': {'id': 160, 'link': 'encdb0', 'mtu': 9000, 'addresses': ['192.168.160.53/24']}, 'encdc0.150': {'id': 150, 'link': 'encdc0', 'mtu': 9000, 'addresses': ['192.168.150.53/24']}, 'enP50s3832.170': {'id': 170, 'link': 'enP50s3832', 'mtu': 9000, 'addresses': ['192.168.170.53/24']}}

DEBUG:Merged config:

network:

  ethernets:

    enP50s3832:

      addresses:

      - 11.111.112.214/22

    enP53p0s0:

      addresses:

      - 11.111.112.215/22

    encdb0:

      addresses:

      - 11.111.114.213/22

      macaddress: 02:76:54:00:00:03

    encdc0:

      addresses:

      - 11.111.112.213/22

      macaddress: 02:76:54:00:00:04

    ence0f:

      addresses:

      - 9.11.116.213/24

      gateway4: 9.11.116.1

      nameservers:

        addresses:

        - 9.11.227.25

  version: 2

  vlans:

    enP50s3832.170:

      addresses:

      - 192.168.170.53/24

      id: 170

      link: enP50s3832

      mtu: 9000

    enP53p0s0.171:

      addresses:

      - 192.168.171.53/24

      id: 171

      link: enP53p0s0

      mtu: 9000

    encdb0.160:

      addresses:

      - 192.168.160.53/24

      id: 160

      link: encdb0

      mtu: 9000

    encdc0.150:

      addresses:

      - 192.168.150.53/24

      id: 150

      link: encdc0

      mtu: 9000

And check the network configuration:
root@ilabg13:~# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: encdb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000

    link/ether 02:76:54:00:00:03 brd ff:ff:ff:ff:ff:ff

    inet 11.111.114.213/22 brd 11.111.115.255 scope global encdb0

       valid_lft forever preferred_lft forever

    inet6 fe80::76:54ff:fe00:3/64 scope link

       valid_lft forever preferred_lft forever

3: ence0f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000

    link/ether 02:76:54:00:00:08 brd ff:ff:ff:ff:ff:ff

    inet 9.11.116.213/24 brd 9.11.116.255 scope global ence0f

       valid_lft forever preferred_lft forever

    inet6 2002:90b:e006:116:76:54ff:fe00:8/64 scope global dynamic mngtmpaddr noprefixroute

       valid_lft 2591895sec preferred_lft 604695sec

    inet6 fe80::76:54ff:fe00:8/64 scope link

       valid_lft forever preferred_lft forever

4: encdc0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000

    link/ether 02:76:54:00:00:04 brd ff:ff:ff:ff:ff:ff

    inet 11.111.112.213/22 brd 11.111.115.255 scope global encdc0

       valid_lft forever preferred_lft forever

    inet6 fe80::76:54ff:fe00:4/64 scope link

       valid_lft forever preferred_lft forever

5: enP50s3832: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000

    link/ether 82:0c:9b:a8:a1:b2 brd ff:ff:ff:ff:ff:ff

    inet 11.111.112.214/22 brd 11.111.115.255 scope global enP50s3832

       valid_lft forever preferred_lft forever

    inet6 fe80::800c:9bff:fea8:a1b2/64 scope link

       valid_lft forever preferred_lft forever

6: enP53p0s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000

    link/ether 82:0c:9b:c9:78:f8 brd ff:ff:ff:ff:ff:ff

    inet 11.111.112.215/22 brd 11.111.115.255 scope global enP53p0s0

       valid_lft forever preferred_lft forever

    inet6 fe80::800c:9bff:fec9:78f8/64 scope link

       valid_lft forever preferred_lft forever

7: enP53p0s0.171@enP53p0s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000

    link/ether 82:0c:9b:c9:78:f8 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::800c:9bff:fec9:78f8/64 scope link

       valid_lft forever preferred_lft forever

8: enP50s3832.170@enP50s3832: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000

    link/ether 82:0c:9b:a8:a1:b2 brd ff:ff:ff:ff:ff:ff

    inet 192.168.170.53/24 brd 192.168.170.255 scope global enP50s3832.170

       valid_lft forever preferred_lft forever

    inet6 fe80::800c:9bff:fea8:a1b2/64 scope link

       valid_lft forever preferred_lft forever

9: encdc0.150@encdc0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000

    link/ether 02:76:54:00:00:04 brd ff:ff:ff:ff:ff:ff

    inet 192.168.150.53/24 brd 192.168.150.255 scope global encdc0.150

       valid_lft forever preferred_lft forever

    inet6 fe80::76:54ff:fe00:4/64 scope link

       valid_lft forever preferred_lft forever

10: encdb0.160@encdb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000

    link/ether 02:76:54:00:00:03 brd ff:ff:ff:ff:ff:ff

    inet 192.168.160.53/24 brd 192.168.160.255 scope global encdb0.160

       valid_lft forever preferred_lft forever

    inet6 fe80::76:54ff:fe00:3/64 scope link

       valid_lft forever preferred_lft forever

The vLAN 171 is not getting the IP assigned

*Additional Instructions for Mario <email address hidden>:
-Post a private note with access information to the machine that the bug is occuring on.

bugproxy (bugproxy)
tags: added: architecture-s3903164 bugnameltc-192830 severity-high targetmilestone-inin---
Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → linux (Ubuntu)
Andrew Cloke (andrew-cloke)
Changed in ubuntu-z-systems:
importance: Undecided → High
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
Frank Heimes (fheimes)
Changed in linux (Ubuntu):
assignee: Skipper Bug Screeners (skipper-screen-team) → nobody
Revision history for this message
Frank Heimes (fheimes) wrote :

Reading through the info provided I found it surprising that the IP address could not be attached to only one of the vlan devices (enP53p0s0.171@enP53p0s0).

Can you share how the devices were activated and how they are based upon? (OSA/qeth, hipersockets/qeth or RoCE/SMC)

I assume that all interfaces that start with "enP*" are based on RoCE/SMC devices, right?

If this is the case I find the naming very strange:
This name seems to be fine: enP53p0s0
But this looks odd to me: enP50s3832 - especially the ending: 3832

On my test system (that unfortunately only has older X-3 Pro based RoCE adapters) the naming is like this (and with that pretty structured and straight-forward):
   enP1p0s0
   enP1p0s0d1
   enP2p0s0
   enP2p0s0d1
So is enP50s3832 really a RoCE interface?

And are there any udev rules that were written or modified to modify the name of this adapter? Or is it otherwise traceable where this name is coming from (even knowing that this is the interface that got the ip address assigned).

Could you also please share the output of 'sudo lshw'?

Changed in ubuntu-z-systems:
status: New → Incomplete
Revision history for this message
Frank Heimes (fheimes) wrote :

I forgot to ask you to please provide the the journal logs, too.

Revision history for this message
bugproxy (bugproxy) wrote : lshw

------- Comment (attachment only) From <email address hidden> 2021-06-01 13:30 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla
Download full text (7.2 KiB)

------- Comment From <email address hidden> 2021-06-01 13:57 EDT-------
This is the actual configuration of HBA assigned to the Host:

root@ilabg13:~# znetconf -c
Device IDs Type Card Type CHPID Drv. Name State
-------------------------------------------------------------------------------------
0.0.0e0f,0.0.0e10,0.0.0e11 1731/01 OSD_1000 EE qeth ence0f online
0.0.0dc0,0.0.0dc1,0.0.0dc2 1731/01 OSD_25GIG DA qeth encdc0 online
0.0.0db0,0.0.0db1,0.0.0db2 1731/01 OSD_25GIG D9 qeth encdb0 online

root@ilabg13:~# lspci
0032:00:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
0035:00:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]

The RoCE adapters are:
enP50s3832 <- working ok
enP53p0s0 <- vLan not getting IP assigned

The full output of lshw is in the attachments, here the reduced output for network class:
root@ilabg13:~# lshw -class network
*-network:0
description: Ethernet interface
product: MT27710 Family [ConnectX-4 Lx Virtual Function]
vendor: Mellanox Technologies
physical id: 5
bus info: pci@0032:00:00.0
logical name: enP50s3832
version: 00
serial: 82:0c:9b:a8:a1:b2
capacity: 25Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pciexpress msix bus_master cap_list ethernet physical fibre 1000bt-fd 10000bt-fd 25000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.0-0 duplex=full firmware=14.25.1020 (IBM0000000016) ip=11.111.112.214 latency=0 link=yes multicast=yes port=fibre
resources: iomemory:ffffd0000-ffffcffff irq:0 memory:8000000000000000-80000000000fffff
*-network:1
description: Ethernet interface
product: MT27710 Family [ConnectX-4 Lx Virtual Function]
vendor: Mellanox Technologies
physical id: 6
bus info: pci@0035:00:00.0
logical name: enP53p0s0
version: 00
serial: 82:0c:9b:c9:78:f8
capacity: 25Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pciexpress msix bus_master cap_list ethernet physical fibre 1000bt-fd 10000bt-fd 25000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.0-0 duplex=full firmware=14.25.1020 (IBM0000000016) ip=11.111.112.215 latency=0 link=yes multicast=yes port=fibre
resources: iomemory:fffffc000-fffffbfff irq:0 memory:8001000000000000-80010000000fffff
*-device:7
description: Ethernet interface
product: 1732/01
physical id: 8
bus info: ccw@0.0.0db0
logical name: encdb0
serial: 02:76:54:00:00:03
capabilities: ethernet physical fibre autonegotiation
configuration: autonegotiation=on broadcast=yes driver=qeth_l2 driverversion=1.0 duplex=full firmware=0165 ip=11.111.114.213 link=yes multicast=yes port=fibre
*-device:8
description: OSA-Express QDIO channel
product: 1732/01
physical id: 9
bus info: ccw@0.0.0db1
configuration: driver=qeth
*-device:9
description: OSA-Express QDIO channel
product: 1732/01
physical id: a
bus info: ccw@0.0.0db2
configuration: driver=qeth
*-device:10
description: Ethernet interface
product: 1732/01
physical id: b
bus info: ccw@0.0.0dc0
logical name: encdc0
serial: 02:76:54:00:00:04
capabilities: ...

Read more...

Revision history for this message
Frank Heimes (fheimes) wrote :

I was now able to work on trying to recreate this situation - even if I will not be able to recreate it exactly due to different CEC and peripherals.

On a clean 20.04.2 install (incl. latest updates) with 4 OSA based qeth devices, I have the first qeth device with only one IP address assigned to the vlan interface - that is what I wanted, since that is the default (and I want to make sure to always be able to reach the system).

Then I added 4 additional qeth devices and assigned IP addresses to the base-, as well as to their vlan-interface, both - everything is fine so far and all 8 (well, in total 9) addresses got statically assigned.

I'll now try to add RoCE interfaces on top of this, but I only have RoCE 1 aka Mellanox Connect-X 3.

Btw. have you been able to gather the journal logs, so that you can share them with us, too?

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2021-06-07 13:56 EDT-------
Could you provide a place to upload the system.journal file? it pass the limit of 51200 KB

Revision history for this message
bugproxy (bugproxy) wrote : system.journal

------- Comment (attachment only) From <email address hidden> 2021-06-07 14:39 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2021-06-07 14:49 EDT-------
Compressed and attached system.journal.

Frank,
I have other systems with similar configurations but not exactly the same. It is another ubuntu host but Native linux installed in its own Lpar, and have some other hosts in a zVM bug with different OS.
All of them with 2 OSA adapters and 2 RoCE.

This is the first time that this issue is seen we have being using this host for at least 6 moths without doing OS reinstallation or changing the adapters and was working correctly.

Revision history for this message
Frank Heimes (fheimes) wrote :

Hi Mario, so did I got it right that this is the only one out of multiple systems with a similar config that has this problem - on a ROcE interface with vlan?
Did you modified expanded the device config of that system over time (with more OSA or more RoCE devices) or was it untouched after initial installation? (Well, I think at least regular updates were done ...)
Are all these systems with RoCE v2.x devices?

What happens if you try to set the IP via the ip command?

Well, I just configured all RoCE devices that I have on this LPAR for vlan usage and assigned IP addresses to the base and vlan interface with the help of the ip command, which worked:
ip a | grep "192.168.1.20"
    inet 192.168.1.201/24 scope global enP1p0s0
    inet 192.168.1.203/24 scope global enP1p0s0d1
    inet 192.168.1.205/24 scope global enP2p0s0
    inet 192.168.1.207/24 scope global enP2p0s0d1
    inet 192.168.1.202/24 scope global enP1p0s0.1234
    inet 192.168.1.204/24 scope global enP1p0s0d1.1234
    inet 192.168.1.206/24 scope global enP2p0s0.1234
    inet 192.168.1.208/24 scope global enP2p0s0d1.1234
Next: transfer this config to the netplan yaml ...

Revision history for this message
bugproxy (bugproxy) wrote : joutnalctl

------- Comment (attachment only) From <email address hidden> 2021-06-07 16:08 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2021-06-07 16:13 EDT-------
Just executed netplan try and netplan apply and take snapshot of journalctl.
It is attached now!

- The configuration has not being touched for long time
- Correct is the only one presenting this problem
- Correct all of the host has RoCe v2.X installed

Could you explain what do you mean with setup the vLan using the IP commands?

Revision history for this message
Frank Heimes (fheimes) wrote :

Oh, I simply meant using the 'ip' command, like (in my example):
$ ip a | grep enP1
6: enP1p0s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000

$ sudo ip link add link enP1p0s0 name enP1p0s0.1234 type vlan id 1234

$ sudo ip addr add 192.168.1.201/24 dev enP1p0s0
$ sudo ip addr add 192.168.1.202/24 dev enP1p0s0.1234

$ sudo ip link set dev enP1p0s0 up
$ sudo ip link set dev enP1p0s0.1234 up

$ ip a

Revision history for this message
Frank Heimes (fheimes) wrote :

So I tried to recreate and provoke any problems with the IP addresses on a test LPAR that has in between 5 OSA based qeth devices (4 of them with two addresses - one on base, one on the vlan device) and 4 RoCE (v1) ports.
Other than finding out that one of the RoCE ports has probably no link (which is a different issue) everything worked fine so far.

And the journal log you provided (thx for that) only shows that the enP53p0s0.171 is not activated (at least for ipv6) after the second systemd network-service start:
...
Jun 07 13:04:05 ilabg13.tuc.stglabs.ibm.com systemd[1]: Starting Network Service...
Jun 07 13:04:05 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2097602]: encdb0.160: Gained IPv6LL
Jun 07 13:04:05 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2097602]: encdc0.150: Gained IPv6LL
Jun 07 13:04:05 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2097602]: enP50s3832.170: Gained IPv6LL
Jun 07 13:04:05 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2097602]: enP53p0s0.171: Gained IPv6LL
Jun 07 13:04:05 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2097602]: enP53p0s0: Gained IPv6LL
Jun 07 13:04:05 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2097602]: enP50s3832: Gained IPv6LL
Jun 07 13:04:05 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2097602]: ence0f: Gained IPv6LL
Jun 07 13:04:05 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2097602]: encdc0: Gained IPv6LL
Jun 07 13:04:05 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2097602]: encdb0: Gained IPv6LL
Jun 07 13:04:05 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2097602]: Enumeration completed
Jun 07 13:04:05 ilabg13.tuc.stglabs.ibm.com systemd[1]: Started Network Service.
Jun 07 13:04:05 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2097602]: encdb0.160: IPv6 successfully enabled
Jun 07 13:04:05 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2097602]: encdc0.150: IPv6 successfully enabled
Jun 07 13:04:05 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2097602]: enP50s3832.170: IPv6 successfully enabled
Jun 07 13:04:05 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2097602]: enP53p0s0: IPv6 successfully enabled
Jun 07 13:04:05 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2097602]: enP50s3832: IPv6 successfully enabled
Jun 07 13:04:05 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2097602]: ence0f: IPv6 successfully enabled
Jun 07 13:04:05 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2097602]: encdc0: IPv6 successfully enabled
Jun 07 13:04:05 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2097602]: encdb0: IPv6 successfully enabled
...
On the first the seems to be some activity.

Unfortunately there are not enough details (in the regular journal log) to dig deeper into this - the above snippet is largely it.
Would it be possible to enable verbose journaling 'journalctl -o verbose' hoping that we are then able to find more details - not sure what it will bring on top (or if this is not sufficient maybe even running systemd in debug mode).

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2021-06-09 05:41 EDT-------
Hi Frank,
Do we know which component is supposed to set the IP address in Mario's case? Probably netplan...?! It's debug log doesn't say anything about IP addresses being set, which is unfortunate.
So here's a couple of oddities(?) / shots in the dark:
(1) Mario's config file has no "renderer" statement - maybe add one?
(2) "version: 2" appears at the end - that's what netplan, according to its log, does anyway - but on my system, that's at the top of the file, not at the end. Maybe move it up? I.e. my config file looks like this:
network:
version: 2
renderer: networkd
ethernets:
(3) enP53p0s0.171 is the final interface listed in the config file. Can you move to a different position to check if that makes a difference?

Revision history for this message
Frank Heimes (fheimes) wrote :

Hi Stefan, the network configuration and the setting of the IP addresses is triggered by netplan and done via systemd-networkd.

Nowadays no explicit renderer is needed anymore, in case there is none, networkd is used as default.
(These days you only need to explicitly specify a renderer if you want to use a different one, which is possible but not supported.)
One may add the "renderer: networkd" key, but I doubt that it will change anything.

The requirement of the "version: 2" key is just that it needs to be on the same level (indention) like ethernets and vlan - which is the case.
Positions should not make a difference (as long as it's under the correct key).

Moving the enP53p0s0.171 should of course not change anything, too - but it's worth a try (since quickly done), just to ensure that there is no issue with the parsing or so.
(One may re-write/re-type this entry on the top, just to make sure that no unusual characters are in - what I do not see, though ...)
Another try would be to test not setting an IP to the base device, so just on the vlan interface only.

Btw. Stefan, what do you think about this 'unusual' name "enP50s3832" of the other RoCE device?
Interestingly this is the one where things work ...
(I saw plans about introducing more predictable names for RoCE interfaces ...)

Revision history for this message
bugproxy (bugproxy) wrote :
Download full text (3.9 KiB)

------- Comment From <email address hidden> 2021-06-09 11:27 EDT-------
Attached the output of journalctl -o verbose

Also did some changes int he netplan config file and the result was the same:

config file:

network:
version: 2
ethernets:
encdb0:
addresses:
- 11.111.114.213/22
macaddress: 02:76:54:00:00:03
encdc0:
addresses:
- 11.111.112.213/22
macaddress: 02:76:54:00:00:04
enP50s3832 :
macaddress: 82:0c:9b:a8:a1:b2
enP53p0s0:
macaddress: 82:0c:9b:c9:78:f8
vlans:
encdb0.160:
id: 160
link: encdb0
mtu: 9000
addresses:
- 192.168.160.53/24
encdc0.150:
id: 150
link: encdc0
mtu: 9000
addresses:
- 192.168.150.53/24
enP53p0s0.171:
id: 171
link: enP53p0s0
mtu: 9000
addresses:
- 192.168.171.53/24
enP50s3832.170:
id: 170
link: enP50s3832
mtu: 9000
addresses:
- 192.168.170.53/24

configuration result:
root@ilabg13:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: encdb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
link/ether 02:76:54:00:00:03 brd ff:ff:ff:ff:ff:ff
inet 11.111.114.213/22 brd 11.111.115.255 scope global encdb0
valid_lft forever preferred_lft forever
inet6 fe80::76:54ff:fe00:3/64 scope link
valid_lft forever preferred_lft forever
3: encdc0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
link/ether 02:76:54:00:00:04 brd ff:ff:ff:ff:ff:ff
inet 11.111.112.213/22 brd 11.111.115.255 scope global encdc0
valid_lft forever preferred_lft forever
inet6 fe80::76:54ff:fe00:4/64 scope link
valid_lft forever preferred_lft forever
4: ence0f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 02:76:54:00:00:08 brd ff:ff:ff:ff:ff:ff
inet 9.11.116.213/24 brd 9.11.116.255 scope global ence0f
valid_lft forever preferred_lft forever
inet6 2002:90b:e006:116:76:54ff:fe00:8/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 2591965sec preferred_lft 604765sec
inet6 fe80::76:54ff:fe00:8/64 scope link
valid_lft forever preferred_lft forever
5: enP50s3832: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
link/ether 82:0c:9b:a8:a1:b2 brd ff:ff:ff:ff:ff:ff
inet6 fe80::800c:9bff:fea8:a1b2/64 scope link
valid_lft forever preferred_lft forever
6: enP53p0s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
link/ether 82:0c:9b:c9:78:f8 brd ff:ff:ff:ff:ff:ff
inet6 fe80::800c:9bff:fec9:78f8/64 scope link
valid_lft forever preferred_lft forever
7: enP53p0s0.171@enP53p0s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 82:0c:9b:c9:78:f8 brd ff:ff:ff:ff:ff:ff
inet6 fe80::800c:9bff:fec9:78f8/64 scope link
valid_lft forever preferred_lft forever
8: enP50s3832.170@enP50s3832: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 82:0c:9b:a8:a1:b2 brd ff:ff:ff:ff:ff:ff
inet 192.168.170.53/24 brd 192.168.170.255 scope global enP50s3832.1...

Read more...

Revision history for this message
bugproxy (bugproxy) wrote : journalctl verbose

------- Comment (attachment only) From <email address hidden> 2021-06-09 11:22 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla
Download full text (3.5 KiB)

------- Comment From <email address hidden> 2021-06-09 11:37 EDT-------
Adding another journalctl to trace the setup with ip command

I proceded to execute:

root@ilabg13:~# ip addr add 192.168.171.53/24 dev enP53p0s0.171

And that get the ip assigned:

root@ilabg13:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: encdb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
link/ether 02:76:54:00:00:03 brd ff:ff:ff:ff:ff:ff
inet 11.111.114.213/22 brd 11.111.115.255 scope global encdb0
valid_lft forever preferred_lft forever
inet6 fe80::76:54ff:fe00:3/64 scope link
valid_lft forever preferred_lft forever
3: encdc0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
link/ether 02:76:54:00:00:04 brd ff:ff:ff:ff:ff:ff
inet 11.111.112.213/22 brd 11.111.115.255 scope global encdc0
valid_lft forever preferred_lft forever
inet6 fe80::76:54ff:fe00:4/64 scope link
valid_lft forever preferred_lft forever
4: ence0f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 02:76:54:00:00:08 brd ff:ff:ff:ff:ff:ff
inet 9.11.116.213/24 brd 9.11.116.255 scope global ence0f
valid_lft forever preferred_lft forever
inet6 2002:90b:e006:116:76:54ff:fe00:8/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 2591957sec preferred_lft 604757sec
inet6 fe80::76:54ff:fe00:8/64 scope link
valid_lft forever preferred_lft forever
5: enP50s3832: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
link/ether 82:0c:9b:a8:a1:b2 brd ff:ff:ff:ff:ff:ff
inet6 fe80::800c:9bff:fea8:a1b2/64 scope link
valid_lft forever preferred_lft forever
6: enP53p0s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
link/ether 82:0c:9b:c9:78:f8 brd ff:ff:ff:ff:ff:ff
inet6 fe80::800c:9bff:fec9:78f8/64 scope link
valid_lft forever preferred_lft forever
7: enP53p0s0.171@enP53p0s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 82:0c:9b:c9:78:f8 brd ff:ff:ff:ff:ff:ff
inet 192.168.171.53/24 scope global enP53p0s0.171
valid_lft forever preferred_lft forever
inet6 fe80::800c:9bff:fec9:78f8/64 scope link
valid_lft forever preferred_lft forever
8: enP50s3832.170@enP50s3832: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 82:0c:9b:a8:a1:b2 brd ff:ff:ff:ff:ff:ff
inet 192.168.170.53/24 brd 192.168.170.255 scope global enP50s3832.170
valid_lft forever preferred_lft forever
inet6 fe80::800c:9bff:fea8:a1b2/64 scope link
valid_lft forever preferred_lft forever
9: encdc0.150@encdc0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 02:76:54:00:00:04 brd ff:ff:ff:ff:ff:ff
inet 192.168.150.53/24 brd 192.168.150.255 scope global encdc0.150
valid_lft forever preferred_lft forever
inet6 fe80::76:54ff:fe00:4/64 scope link
valid_lft forever pr...

Read more...

Revision history for this message
bugproxy (bugproxy) wrote : journalctl vlan setup whit ip cmd

------- Comment (attachment only) From <email address hidden> 2021-06-09 11:34 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2021-06-09 11:50 EDT-------
Did another experiment and removed the IP assigned to both RoCE adapters vLans using ip command and ran the netplan apply.

The issue was recreated only one of the vLans gets the ip assigned the other one didn't.

Revision history for this message
Frank Heimes (fheimes) wrote :

Hi Mario, thanks for the additional testing and logs.

Ok, that the modification of the yaml file will not have an impact was kind of expected, but it was to make sure that the parsing itself is fine - since netplan try and apply comes back w/o an error, everything else would have surprised me.

But interesting is that it works for you using the ip command.
That makes me think that it's probably an issue with systemd-networkd.

So please could you share the output of:
networkctl status enP53p0s0.171 --no-pager
after the interface got an IP address assigned (so after 'ip addr add ...')
and again after such an un-successful netplan apply (so when an address did not got assigned).

And while doing that I recommend to update the systemd-networkd loglevel to debug with the following so that we get as much details as possible:
$ cat /etc/systemd/system/systemd-networkd.service.d/override.conf
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

$ systemctl restart systemd-networkd

I'll also ave a look at systemd upstream (https://github.com/systemd/systemd/issues/), checking if s/t similar got already reported ...

Revision history for this message
bugproxy (bugproxy) wrote :
Download full text (4.1 KiB)

------- Comment From <email address hidden> 2021-06-09 15:04 EDT-------
Steeps followed...

Updated systemd-networkd loglevel:
root@ilabg13:~# cat /etc/systemd/system/systemd-networkd.service.d/override.conf
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug
root@ilabg13:~# systemctl restart systemd-networkd

Removed manually the IP:
ip addr del 192.168.171.53/24 dev enP53p0s0.171

Configure network using netplan:
root@ilabg13:~# netplan apply
root@ilabg13:~# networkctl status enP53p0s0.171 --no-pager
? 7: enP53p0s0.171
Link File: n/a
Network File: n/a
Type: vlan
State: degraded (pending)
HW Address: 82:0c:9b:c9:78:f8
MTU: 9000 (max: 65535)
Queue Length (Tx/Rx): 1/1
Auto negotiation: yes
Speed: 25Gbps
Duplex: full
Port: fibre
Address: fe80::800c:9bff:fec9:78f8

Jun 09 11:59:40 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2801696]: enP53p0s0.171: Flags change: +UP +LOWER_UP +RUNNING +MULTICAST +BROADCAST
Jun 09 11:59:40 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2801696]: enP53p0s0.171: Link 7 added
Jun 09 11:59:40 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2801696]: enP53p0s0.171: link pending udev initialization...
Jun 09 11:59:40 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2801696]: enP53p0s0.171: netdev has index 7
Jun 09 11:59:40 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2801696]: enP53p0s0.171: Saved original MTU: 9000
Jun 09 11:59:40 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2801696]: enP53p0s0.171: Remembering foreign address: fe80::800c:9bff:fec9:78f8/64 (valid forever)
Jun 09 11:59:40 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2801696]: enP53p0s0.171: Gained IPv6LL
Jun 09 11:59:40 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2801696]: enP53p0s0.171: Remembering route: dst: ff00::/8, src: n/a, gw: n/a, prefsrc: n/a, scope: glob? multicast
Jun 09 11:59:40 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2801696]: enP53p0s0.171: Remembering route: dst: fe80::800c:9bff:fec9:78f8/128, src: n/a, gw: n/a, pref?ype: local
Jun 09 11:59:40 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2801696]: enP53p0s0.171: Remembering route: dst: fe80::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: glo?e: unicast

Manually adding IP:
root@ilabg13:~# ip addr add 192.168.171.53/24 dev enP53p0s0.171
root@ilabg13:~# networkctl status enP53p0s0.171 --no-pager
? 7: enP53p0s0.171
Link File: n/a
Network File: n/a
Type: vlan
State: routable (pending)
HW Address: 82:0c:9b:c9:78:f8
MTU: 9000 (max: 65535)
Queue Length (Tx/Rx): 1/1
Auto negotiation: yes
Speed: 25Gbps
Duplex: full
Port: fibre
Address: 192.168.171.53
fe80::800c:9bff:fec9:78f8

Jun 09 11:59:40 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2801696]: enP53p0s0.171: Remembering foreign address: fe80::800c:9bff:fec9:78f8/64 (valid forever)
Jun 09 11:59:40 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2801696]: enP53p0s0.171: Gained IPv6LL
Jun 09 11:59:40 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2801696]: enP53p0s0.171: Remembering route: dst: ff00::/8, src: n/a, gw: n/a, prefsrc: n/a, scope: glob? multicast
Jun 09 11:59:40 ilabg13.tuc.stglabs.ibm.com systemd-networkd[2801696]: enP53p0s0.171: Remembering route: dst: fe80::800c:9bff:fec9:78f8/128, src: n/a, gw: n/a, pref?ype: local
Jun 09 11:59:40 ilab...

Read more...

Revision history for this message
Frank Heimes (fheimes) wrote :

Are there now more details after:
systemctl daemon-reload && systemctl restart systemd-networkd
provided by:
journalctl -b -u systemd-networkd --no-pager
?

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2021-06-09 17:17 EDT-------
Attaching output of 'journalctl -b -u systemd-networkd --no-pager' after 'systemctl daemon-reload && systemctl restart systemd-networkd' execution

File -> journalctl_deamon-reload

Revision history for this message
bugproxy (bugproxy) wrote : journalctl_deamon-reload

------- Comment (attachment only) From <email address hidden> 2021-06-09 17:14 EDT-------

Revision history for this message
Frank Heimes (fheimes) wrote :

Ok, the line that concerns me is:
"
enP53p0s0.171: link pending udev initialization...
"

It looks like systemd-networkd thinks that udev is not done with enP53p0s0.171 --> "pending", but configuring it with ip works - so could be networkd or also udevd.

So lets expand the journal output with the udev messages and share it again:
journalctl -b -u systemd-networkd -u systemd-udevd --no-pager

Please can you also share the output of:
apt-cache policy systemd udev netplan.io

Investigations pointed me to:
https://github.com/systemd/systemd/issues/15445#issuecomment-776856604
(changing the interface name at some point could be interesting, too...)

Anyway, I'll mark this bug as affecting systemd now.

Revision history for this message
Frank Heimes (fheimes) wrote :

I should have added that it would be best in this case to have the debug mode enabled for udevd, too:
$ cat /etc/systemd/system/systemd-udevd.service.d/override.conf
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug
$ sudo systemctl daemon-reload && sudo systemctl restart systemd-udevd
$ journalctl -b -u systemd-networkd -u systemd-udevd --no-pager
(instead of -b you may do '--since="today"' and/or '--until="5 minutes ago"' or so to reduce the output)

You may then trigger the udev rules directly, like:
$ sudo udevadm trigger
and once indirectly via netplan (and maybe via 'ip').

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2021-06-10 03:55 EDT-------
Makes me wonder if what we're seeing is a side-effect of

------- Comment From <email address hidden> 2021-06-10 03:58 EDT-------
Makes me wonder whether this could be a side-effect of the interface name, indeed.
@Viktor/Niklas: Interface enP50s3832.170 gets an IP assigned, no problem. But enP53p0s0.171 does not, and Frank noticed that systemd spills out this message:

enP53p0s0.171: link pending udev initialization...

So could it be that systemd trips over the FID containing an alphabetic character, and never registers the interface as properly initialized..?

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2021-06-11 10:04 EDT-------
Talked to Viktor - he didn't think there's much indication that the skewed interface name has any effect. Especially since the interface gets a LL address assigned just fine.

bugproxy (bugproxy)
tags: added: targetmilestone-inin2004
removed: targetmilestone-inin---
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2021-06-14 11:18 EDT-------
Enable udev debug mode:
root@ilabg13:~# cat /etc/systemd/system/systemd-udevd.service.d/override.conf
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug
root@ilabg13:~# systemctl daemon-reload && sudo systemctl restart systemd-udevd

Details about apt-cache policy systemd udev netplan.io:
root@ilabg13:~# apt-cache policy systemd udev netplan.io
systemd:
Installed: 245.4-4ubuntu3.6
Candidate: 245.4-4ubuntu3.6
Version table:
*** 245.4-4ubuntu3.6 500
500 http://us.ports.ubuntu.com/ubuntu-ports focal-updates/main s390x Packages
100 /var/lib/dpkg/status
245.4-4ubuntu3 500
500 http://us.ports.ubuntu.com/ubuntu-ports focal/main s390x Packages
udev:
Installed: 245.4-4ubuntu3.6
Candidate: 245.4-4ubuntu3.6
Version table:
*** 245.4-4ubuntu3.6 500
500 http://us.ports.ubuntu.com/ubuntu-ports focal-updates/main s390x Packages
100 /var/lib/dpkg/status
245.4-4ubuntu3 500
500 http://us.ports.ubuntu.com/ubuntu-ports focal/main s390x Packages
netplan.io:
Installed: 0.102-0ubuntu1~20.04.2
Candidate: 0.102-0ubuntu1~20.04.2
Version table:
*** 0.102-0ubuntu1~20.04.2 500
500 http://us.ports.ubuntu.com/ubuntu-ports focal-updates/main s390x Packages
100 /var/lib/dpkg/status
0.99-0ubuntu1 500
500 http://us.ports.ubuntu.com/ubuntu-ports focal/main s390x Packages

Steeps followed to retrieve logs:
- Manually remove the IP
ip addr add 192.168.171.53/24 dev enP53p0s0.171
- Try to assign ip via netplan
root@ilabg13:~# date
Mon Jun 14 08:12:02 MST 2021
root@ilabg13:~# netplan apply
root@ilabg13:~# date
Mon Jun 14 08:12:12 MST 2021
- Try to assign ip via udevadm
root@ilabg13:~# date
Mon Jun 14 08:12:28 MST 2021
root@ilabg13:~# udevadm trigger
root@ilabg13:~# date
Mon Jun 14 08:12:36 MST 2021
- Try to assign ip via ip command (Successfully assigned)
root@ilabg13:~# date
Mon Jun 14 08:13:02 MST 2021
root@ilabg13:~# ip addr add 192.168.171.53/24 dev enP53p0s0.171
RTNETLINK answers: File exists
root@ilabg13:~# date
Mon Jun 14 08:13:07 MST 2021
- Retrieve journalctl logs
root@ilabg13:~# journalctl --since today -u systemd-networkd -u systemd-udevd --no-pager > journalctl_udevadm_netplan_ip.txt

Revision history for this message
bugproxy (bugproxy) wrote : journalctl udevadm netplan ip

------- Comment (attachment only) From <email address hidden> 2021-06-14 11:21 EDT-------

Revision history for this message
Frank Heimes (fheimes) wrote :

Thanks Mario the output is helpful, especially in combination with the time indices.

So udev does not seem to be able to complete the initialization of enP53p0s0.171:
"
Jun 14 08:12:10 ilabg13.tuc.stglabs.ibm.com systemd-networkd[4102292]: enP53p0s0.171: link pending udev initialization...
"
Would you mind having a look at:
$ ls -l /run/systemd/network/
and share the _entire_ content of this folder?
(Especially the files related to 'enP53p0s0', but for comparison reasons ideally all (network interface) files.)
I want to compare the content with the config in the netplan yaml file.
(That would hopefully allow us to check if then netplan config got properly translated to systemd network files.)

On top could you also try to bring up the ip address (again successfully and unsuccessfully) with having the "sudo udevadm --debug monitor --property" active?
(also triggering network only like this: 'sudo udevadm trigger --attr-match=subsystem=net')

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2021-06-15 11:18 EDT-------
Enable udev monitor:
udevadm --debug monitor --property >> udevadm_monitor.txt

Steeps followed to retrieve logs:
- Manually remove the IP
ip addr del 192.168.171.53/24 dev enP53p0s0.171

- Try to assign ip via netplan
root@ilabg13:~# date
Tue Jun 15 08:10:29 MST 2021
root@ilabg13:~# netplan apply
root@ilabg13:~# date
Tue Jun 15 08:10:59 MST 2021

- Try to assign ip via udevadm
root@ilabg13:~# date
Tue Jun 15 08:11:21 MST 2021
root@ilabg13:~# udevadm trigger --attr-match=subsystem=net
root@ilabg13:~# date
Tue Jun 15 08:11:31 MST 2021

- Try to assign ip via ip command (Successfully assigned)
root@ilabg13:~# date
Tue Jun 15 08:11:46 MST 2021
root@ilabg13:~# ip addr add 192.168.171.53/24 dev enP53p0s0.171
RTNETLINK answers: File exists
root@ilabg13:~# date
Tue Jun 15 08:11:51 MST 2021

- Retrieve journalctl logs
journalctl --since today -u systemd-networkd -u systemd-udevd --no-pager > journalctl_udev_monitor.txt

Copied the full content of /run/systemd/network

Revision history for this message
bugproxy (bugproxy) wrote : udevadm monitor

------- Comment (attachment only) From <email address hidden> 2021-06-15 11:48 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : journalctl udev monitor

------- Comment (attachment only) From <email address hidden> 2021-06-15 11:49 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : network

------- Comment (attachment only) From <email address hidden> 2021-06-15 11:50 EDT-------

Frank Heimes (fheimes)
Changed in systemd (Ubuntu):
assignee: nobody → Canonical Foundations Team (canonical-foundations)
Matthieu Clemenceau (mclemenceau)
tags: added: fr-1459
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2021-06-17 12:14 EDT-------
Right now the system is at kernel version:
root@ilabg13:~# uname -a
Linux ilabg13.tuc.stglabs.ibm.com 5.4.0-73-generic #82-Ubuntu SMP Wed Apr 14 17:29:32 UTC 2021 s390x s390x s390x GNU/Linux

And it is a new version available:
5.4.0-74-generic

It is recommended to upgrade or do you need any more info before doing the update?

Revision history for this message
Frank Heimes (fheimes) wrote :

No, I think we cannot gather much more data and logs than we already did.
(Except the ude rules themselves, but they will not change with a kernel update.)

I am relatively confident that this is not an netplan issue since I verified the files in /run/systemd/network/ and the look good and fit nicely to the yaml.
I found a similar upstream issue, checking if this is close enough.
I'm gonna reach out to another team and will discuss some potential next steps.

So you can (and should) update, since the 5.4.0-74 is a significant update (and incl. the upstream stable patches from v5.4.107 to .114).
And please watch out for any behavioral changes regarding the udev rules and the address assignments!

Revision history for this message
Lukas Märdian (slyon) wrote :
Download full text (10.7 KiB)

After crushing the logs a bit more, I found a few interesting things and also have a few open questions. Thank you very much for your `date` timestamps in between the different steps, those are very helpful in linking the order of events!

Open questions:
1) The "udevadm-monitor.txt" log shows that there is a manual "/etc/systemd/network/10-enP53p0s0.link" configuration file, while all other interfaces make use of "/usr/lib/systemd/network/99-default.link". Can you show us what is inside this file?

2) In comment #22 you show the output of "networkctl":
root@ilabg13:~# netplan apply
root@ilabg13:~# networkctl status enP53p0s0.171 --no-pager
? 7: enP53p0s0.171
Link File: n/a
Network File: n/a
Type: vlan
State: degraded (pending)
HW Address: 82:0c:9b:c9:78:f8

=> I wonder why "Link File: n/a" and "Network File: n/a" is set.. The Link state is still "(pending)", would you mind executing the same steps again and re-executing the "networkctl status enP53p0s0.171 --no-pager" a bit later again (like 10sec after and 30sec after)?

3) Does the workaround from https://github.com/systemd/systemd/issues/15445#issuecomment-776856604 work for you? E.g.:

# netplan apply
- wait 15 sec
# networkctl reload
- wait 15 sec
# networkctl renew enP53p0s0.171
- wait 15 sec
# networkctl reconfigure enP53p0s0.171
- wait 15 sec
# networkctl reconfigure enP53p0s0.171
- wait 30 sec
# networkctl status enP53p0s0.171 --no-pager
# ip a

=== Findings from comment #33 (journalctl_udev_monitor.txt) ===
You executed "netplan apply" between 08:10:29 - 08:10:59 and it looks like systemd-networkd found the correct netplan-*.network configuration and applied it (assigned the IP) at 08:10:57. See "***" lines:

Jun 15 08:10:57 systemd-networkd[182847]: enP53p0s0.171: Link state is up-to-date
*** Jun 15 08:10:57 systemd-networkd[182847]: enP53p0s0.171: found matching network '/run/systemd/network/10-netplan-enP53p0s0.171.network'
Jun 15 08:10:57 systemd-networkd[182847]: Setting '/proc/sys/net/ipv6/conf/enP53p0s0.171/disable_ipv6' to '0'
Jun 15 08:10:57 systemd-networkd[182847]: enP53p0s0.171: IPv6 successfully enabled
Jun 15 08:10:57 systemd-networkd[182847]: Setting '/proc/sys/net/ipv6/conf/enP53p0s0.171/proxy_ndp' to '0'
Jun 15 08:10:57 systemd-networkd[182847]: Setting '/proc/sys/net/ipv6/conf/enP53p0s0.171/use_tempaddr' to '0'
Jun 15 08:10:57 systemd-networkd[182847]: Setting '/proc/sys/net/ipv6/conf/enP53p0s0.171/accept_ra' to '0'
Jun 15 08:10:57 systemd-networkd[182847]: LLDP: Started LLDP client
Jun 15 08:10:57 systemd-networkd[182847]: enP53p0s0.171: Started LLDP.
Jun 15 08:10:57 systemd-networkd[182847]: enP53p0s0.171: Setting address genmode for link
[...]
*** Jun 15 08:10:57 systemd-networkd[182847]: enP53p0s0.171: Remembering updated address: 192.168.171.53/24 (valid forever)
Jun 15 08:10:57 systemd-networkd[182847]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1/link/_37 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=45 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Jun 15 08:10:57 systemd-networkd[182847]: enP53p0s0.171: Remembering route: dst: 192.168.171.53/32, src: n/a, gw: n/a, pref...

Revision history for this message
bugproxy (bugproxy) wrote :
Download full text (10.3 KiB)

------- Comment From <email address hidden> 2021-06-23 18:19 EDT-------
Files content:
root@ilabg13:~# cat /etc/systemd/network/10-enP53p0s0.link
[Match]
MACAddress=82:0c:9b:c9:78:f8

[Link]
Name=enP53p0s0
root@ilabg13:~# cat /usr/lib/systemd/network/99-default.link
# SPDX-License-Identifier: LGPL-2.1+
#
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.

[Match]
OriginalName=*

[Link]
NamePolicy=keep kernel database onboard slot path
AlternativeNamesPolicy=database onboard slot path
MACAddressPolicy=persistent

Steeps followed to retrieve logs:
- Manually remove the IP
root@ilabg13:~# date
Wed Jun 23 15:08:41 MST 2021
root@ilabg13:~# ip addr del 192.168.171.53/24 dev enP53p0s0.171

- Try to assign ip via netplan
root@ilabg13:~# date
Wed Jun 23 15:10:14 MST 2021
root@ilabg13:~# netplan apply

- Checking networkctl 10sec and 30sec after
root@ilabg13:~# date
Wed Jun 23 15:11:19 MST 2021
root@ilabg13:~# networkctl status enP53p0s0.171 --no-pager
? 9: enP53p0s0.171
Link File: n/a
Network File: n/a
Type: vlan
State: degraded (pending)
HW Address: 82:0c:9b:c9:78:f8
MTU: 9000 (max: 65535)
Queue Length (Tx/Rx): 1/1
Auto negotiation: yes
Speed: 25Gbps
Duplex: full
Address: fe80::800c:9bff:fec9:78f8
Activation Policy: up

Jun 23 15:10:25 ilabg13.tuc.stglabs.ibm.com systemd-networkd[9805]: enP53p0s0.171: Flags change: +UP +LOWER_UP +RUNNING +MULTICAST +BROADCAST
Jun 23 15:10:25 ilabg13.tuc.stglabs.ibm.com systemd-networkd[9805]: enP53p0s0.171: Link 9 added
Jun 23 15:10:25 ilabg13.tuc.stglabs.ibm.com systemd-networkd[9805]: enP53p0s0.171: link pending udev initialization...
Jun 23 15:10:25 ilabg13.tuc.stglabs.ibm.com systemd-networkd[9805]: enP53p0s0.171: netdev has index 9
Jun 23 15:10:25 ilabg13.tuc.stglabs.ibm.com systemd-networkd[9805]: enP53p0s0.171: Saved original MTU: 9000
Jun 23 15:10:25 ilabg13.tuc.stglabs.ibm.com systemd-networkd[9805]: enP53p0s0.171: Remembering foreign address: fe80::800c:9bff:fec9:78f8/64 (valid forever)
Jun 23 15:10:25 ilabg13.tuc.stglabs.ibm.com systemd-networkd[9805]: enP53p0s0.171: Gained IPv6LL
Jun 23 15:10:25 ilabg13.tuc.stglabs.ibm.com systemd-networkd[9805]: enP53p0s0.171: Remembering route: dst: ff00::/8, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: loc?e: multicast
Jun 23 15:10:25 ilabg13.tuc.stglabs.ibm.com systemd-networkd[9805]: enP53p0s0.171: Remembering route: dst: fe80::800c:9bff:fec9:78f8/128, src: n/a, gw: n/a, prefsrc: n/a, scop? type: local
Jun 23 15:10:25 ilabg13.tuc.stglabs.ibm.com systemd-networkd[9805]: enP53p0s0.171: Remembering route: dst: fe80::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: ma?ype: unicast
root@ilabg13:~#
root@ilabg13:~#
root@ilabg13:~# date
Wed Jun 23 15:11:44 MST 2021
root@ilabg13:~# networkctl status enP53p0s0.171 --no-pager
? 9: enP53p0s0.171
Link File: n/a
Network File: n/a
Type: vlan
State: degraded (pending)
HW Address: 82:0c:9b:c9:78:f8
MTU: 9000 (max: 65535)
Queue Length (Tx/Rx): 1/1
Auto negotiation: yes
S...

Revision history for this message
bugproxy (bugproxy) wrote : journalctl delay

------- Comment (attachment only) From <email address hidden> 2021-06-23 18:25 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : udevadm monitor delay

------- Comment (attachment only) From <email address hidden> 2021-06-23 18:26 EDT-------

Revision history for this message
Lukas Märdian (slyon) wrote :

> Gave some time between netplan apply and check without success, I can leave it for
> hours and the IP never gets assigned

Indeed, I cannot see it being assigned an IPv4 address in your latest "_delay" run. (Well, there is some assignment of an IP to enP53p0s0.171 at 15:05:29 in the journal logs, but that was before you started the experiment).

> maybe what you are seeing is the manual assignment that I did with
> ip addr add 192.168.171.53/24 dev enP53p0s0.171

No I don't think so, as the timestamp of your manual "ip addr add" command (08:13:02++) is after the systemd-networkd assignment in the logs (08:12:38). Also, your output clearly shows that the IP was already assigned when you tried to (re-)assign it manually (it shows the "RTNETLINK answers: File exists" error):

> root@ilabg13:~# ip addr add 192.168.171.53/24 dev enP53p0s0.171
> RTNETLINK answers: File exists

I'm a bit worried about the /etc/systemd/network/10-enP53p0s0.link file (now there seems to be another one "10-enP53p0s0.link" as well...) – Is this file needed? Why was it created?
It matches on "MACAddress=82:0c:9b:c9:78:f8", which would match enP53p0s0 AND enP53p0s0.171 at the same time (they use the same inherited MAC address), leading to unspecified behavior, as we cannot rename two interfaces to the same "Name=enP53p0s0".

Could you try deleting this file in /etc/systemd/network/10-enP53p0s0.link (potentially adopting your netplan YAML config to the name assigned to this interface by the kernel/systemd) and check if this makes any difference?
# rm /etc/systemd/network/10-enP53p0s0.link

If for some reason you cannot delete this file, try adding a "Type=!vlan" line instead:
# cat /etc/systemd/network/10-enP53p0s0.link
[Match]
MACAddress=82:0c:9b:c9:78:f8
Type=!vlan

[Link]
Name=enP53p0s0

Does this make any difference when running "netplan apply && sleep 30 && networkctl status -a --no-pager"?

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla
Download full text (5.7 KiB)

------- Comment From <email address hidden> 2021-06-24 11:53 EDT-------
Giving it a try without the /etc/systemd/network/ files

- Checking that it is empty
root@ilabg13:~# ll /etc/systemd/network/
total 8
drwxr-xr-x 2 root root 4096 Jun 24 08:41 ./
drwxr-xr-x 5 root root 4096 Jun 17 06:18 ../

- Try to assign ip via netplan
root@ilabg13:~# date
Thu Jun 24 08:47:45 MST 2021
root@ilabg13:~# netplan apply

- Checking networkctl
root@ilabg13:~# date
Thu Jun 24 08:49:10 MST 2021
root@ilabg13:~# networkctl status enP53p0s0.171 --no-pager
? 9: enP53p0s0.171
Link File: n/a
Network File: n/a
Type: vlan
State: degraded (pending)
HW Address: 82:0c:9b:c9:78:f8
MTU: 9000 (max: 65535)
Queue Length (Tx/Rx): 1/1
Auto negotiation: yes
Speed: 25Gbps
Duplex: full
Address: fe80::800c:9bff:fec9:78f8
Activation Policy: up

Jun 24 08:48:01 ilabg13.tuc.stglabs.ibm.com systemd-networkd[293700]: enP53p0s0.171: Flags change: +UP +LOWER_UP +RUNNING +MULTICAST +BROADCAST
Jun 24 08:48:01 ilabg13.tuc.stglabs.ibm.com systemd-networkd[293700]: enP53p0s0.171: Link 9 added
Jun 24 08:48:01 ilabg13.tuc.stglabs.ibm.com systemd-networkd[293700]: enP53p0s0.171: link pending udev initialization...
Jun 24 08:48:01 ilabg13.tuc.stglabs.ibm.com systemd-networkd[293700]: enP53p0s0.171: netdev has index 9
Jun 24 08:48:01 ilabg13.tuc.stglabs.ibm.com systemd-networkd[293700]: enP53p0s0.171: Saved original MTU: 9000
Jun 24 08:48:01 ilabg13.tuc.stglabs.ibm.com systemd-networkd[293700]: enP53p0s0.171: Remembering foreign address: fe80::800c:9bff:fec9:78f8/64 (valid forever)
Jun 24 08:48:01 ilabg13.tuc.stglabs.ibm.com systemd-networkd[293700]: enP53p0s0.171: Gained IPv6LL
Jun 24 08:48:01 ilabg13.tuc.stglabs.ibm.com systemd-networkd[293700]: enP53p0s0.171: Remembering route: dst: ff00::/8, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: local, ?e: multicast
Jun 24 08:48:01 ilabg13.tuc.stglabs.ibm.com systemd-networkd[293700]: enP53p0s0.171: Remembering route: dst: fe80::800c:9bff:fec9:78f8/128, src: n/a, gw: n/a, prefsrc: n/a, scope: g? type: local
Jun 24 08:48:01 ilabg13.tuc.stglabs.ibm.com systemd-networkd[293700]: enP53p0s0.171: Remembering route: dst: fe80::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main, ?ype: unicast

- IP still not being assigned
root@ilabg13:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: encdb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
link/ether 02:76:54:00:00:03 brd ff:ff:ff:ff:ff:ff
inet 11.111.114.213/22 brd 11.111.115.255 scope global encdb0
valid_lft forever preferred_lft forever
inet6 fe80::76:54ff:fe00:3/64 scope link
valid_lft forever preferred_lft forever
3: ence0f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 02:76:54:00:00:04 brd ff:ff:ff:ff:ff:ff
inet 9.11.116.213/24 brd 9.11.116.255 scope global ence0f
valid_lft forever preferred_lft forever
inet6 2002:90b:e006:116:76:54ff:fe00:4/64 scope gl...

Read more...

Revision history for this message
bugproxy (bugproxy) wrote : udevadm monitor nofiles

------- Comment (attachment only) From <email address hidden> 2021-06-24 11:57 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : journalctl no files

------- Comment (attachment only) From <email address hidden> 2021-06-24 11:57 EDT-------

Revision history for this message
Lukas Märdian (slyon) wrote :

Thank you! The core problem still is that systemd-networkd does not find any .link and .network file for the enP53p0s0.171 interface, that's why it cannot apply any configuration (IP address):

> ? 9: enP53p0s0.171
> Link File: n/a
> Network File: n/a

This might very well be related to the additional /etc/systemd/network/10-enP53p0s0.link, which is now gone and apparently not needed (good!) – leave that away.
Still the configuration/match of that file might still be loaded into the running udev, so we must somehow reset udev to a clean state. I'd like to ask you for another test run (and potential reboot of that machine – if that is possible?):

# try to reload udev at runtime
udevadm control --reload; udevadm trigger; udevadm settle
# wait a bit
sleep 30
# apply netplan configuration
netplan --debug apply
# wait a bit
sleep 30
# check results
networkctl status enP53p0s0.171 --no-pager
ip a

If the IP is still not set, could you please try to reboot the machine and check "networkctl" afterwards?

Revision history for this message
Frank Heimes (fheimes) wrote :

Before following Lukas' suggestions could you quickly do a:
   stat /etc/systemd/network/10-enP53p0s0.link
(before deleting this file)
since this would give us the creation / modification time
and potentially (with some log correlation)
this may provide a hint when and why it was created.

Revision history for this message
Frank Heimes (fheimes) wrote :

@Mario Did you had the chance to get the file stats of 10-enP53p0s0.link and to try what Lukas suggested in comment #47?

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2021-07-27 10:40 EDT-------
Since the file /etc/systemd/network/10-enP53p0s0.link was removed it wasn't get generated again.

Is there a way to get it generated again?

Revision history for this message
bugproxy (bugproxy) wrote :
Download full text (15.2 KiB)

------- Comment From <email address hidden> 2021-07-27 11:33 EDT-------
Enable udev monitor:
udevadm --debug monitor --property >> udevadm_reload.txt

Steeps followed to retrieve logs:

- Checking actual status of the vLan:
root@ilabg13:~# date
Tue Jul 27 08:23:47 MST 2021
root@ilabg13:~# ip addr show dev enP53p0s0.171
10: enP53p0s0.171@enP53p0s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 82:0c:9b:c9:78:f8 brd ff:ff:ff:ff:ff:ff
inet6 fe80::800c:9bff:fec9:78f8/64 scope link
valid_lft forever preferred_lft forever

- Reload udevadm
root@ilabg13:~# date
Tue Jul 27 08:24:22 MST 2021
root@ilabg13:~# udevadm control --reload; udevadm trigger; udevadm settle

- Try to assign ip via netplan
root@ilabg13:~# date
Tue Jul 27 08:26:01 MST 2021
root@ilabg13:~# netplan --debug apply
** (generate:3616109): DEBUG: 08:26:04.430: Processing input file /etc/netplan/00-installer-config.yaml..
** (generate:3616109): DEBUG: 08:26:04.431: starting new processing pass
** (generate:3616109): DEBUG: 08:26:04.431: Processing input file /etc/netplan/01-iscsi-config.yaml..
** (generate:3616109): DEBUG: 08:26:04.431: starting new processing pass
** (generate:3616109): DEBUG: 08:26:04.431: We have some netdefs, pass them through a final round of validation
** (generate:3616109): DEBUG: 08:26:04.431: encdc0: setting default backend to 1
** (generate:3616109): DEBUG: 08:26:04.431: Configuration is valid
** (generate:3616109): DEBUG: 08:26:04.431: encdb0: setting default backend to 1
** (generate:3616109): DEBUG: 08:26:04.431: Configuration is valid
** (generate:3616109): DEBUG: 08:26:04.431: ence0f: setting default backend to 1
** (generate:3616109): DEBUG: 08:26:04.431: Configuration is valid
** (generate:3616109): DEBUG: 08:26:04.431: encdb0.160: setting default backend to 1
** (generate:3616109): DEBUG: 08:26:04.431: Configuration is valid
** (generate:3616109): DEBUG: 08:26:04.431: enP50s3832: setting default backend to 1
** (generate:3616109): DEBUG: 08:26:04.431: Configuration is valid
** (generate:3616109): DEBUG: 08:26:04.431: enP53p0s0.171: setting default backend to 1
** (generate:3616109): DEBUG: 08:26:04.431: Configuration is valid
** (generate:3616109): DEBUG: 08:26:04.431: enP50s3832.170: setting default backend to 1
** (generate:3616109): DEBUG: 08:26:04.431: Configuration is valid
** (generate:3616109): DEBUG: 08:26:04.431: enP53p0s0: setting default backend to 1
** (generate:3616109): DEBUG: 08:26:04.431: Configuration is valid
** (generate:3616109): DEBUG: 08:26:04.431: encdc0.150: setting default backend to 1
** (generate:3616109): DEBUG: 08:26:04.431: Configuration is valid
** (generate:3616109): DEBUG: 08:26:04.431: Generating output files..
** (generate:3616109): DEBUG: 08:26:04.431: openvswitch: definition ence0f is not for us (backend 1)
** (generate:3616109): DEBUG: 08:26:04.431: NetworkManager: definition ence0f is not for us (backend 1)
** (generate:3616109): DEBUG: 08:26:04.431: openvswitch: definition encdb0 is not for us (backend 1)
** (generate:3616109): DEBUG: 08:26:04.431: NetworkManager: definition encdb0 is not for us (backend 1)
** (generate:3616109): DEBUG: 08:26:04.432: op...

Revision history for this message
bugproxy (bugproxy) wrote : udevadm_reload

------- Comment (attachment only) From <email address hidden> 2021-07-27 11:37 EDT-------

Revision history for this message
Frank Heimes (fheimes) wrote :

Hi Mario, thanks for coming back to this case.
So it seems that the situation is now good again - now that the /etc/systemd/network/10-enP53p0s0.link got removed.

We don't know a tool or application offhand that would create such a .link file in /etc/systemd/network/ alone.
However, there are plenty of tools that generate files in this context, like netplan, udev, NetworkManager, etc. - but then such automatically created files would be usually located at /run/systemd/network - /etc is more for manually created ones.

Maybe this was caused by a less common tool that does not stick to this (de-facto) standard.
Or maybe it got copied there by accident, while backing up files or so - difficult to say.
We at least don't imagine a case where such a file is created by the standard Ubuntu tooling.

You didn't had a chance to grab the stats for that file before having removed it, did you?

Do you agree that the bug can be closed now based on finding the cause (the .link) file, having it removed and now the system behaving as expected again?

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2021-07-28 17:41 EDT-------
Sound good for me, lets close it

Frank Heimes (fheimes)
Changed in linux (Ubuntu):
status: New → Invalid
Changed in netplan.io (Ubuntu):
status: New → Invalid
Changed in systemd (Ubuntu):
status: New → Invalid
Changed in ubuntu-z-systems:
status: Incomplete → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.