netplan not sending DHCP request of IPoIB interfaces.

[jwiegley]

netplan for infiniband does not send out any DHCP requests. It works fine for other Ethernet interfaces. if I manually just issue 'dhclient ibs2' is gets a DHCP address just fine.

Somehow, networkd doesn't invoke a DHCP client for infiniband networks.

root@hph7a:~# more /etc/netplan/01-netcfg.yaml
# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
  version: 2
  renderer: networkd
  ethernets:
    enp2s0f0:
      dhcp4: yes
    enp2s0f1:
      dhcp4: yes
    ibs2:
      dhcp4: yes
      dhcp6: no
    ibs2d1:
      dhcp4: yes

after booting or manually running 'netplan apply' this is what the interfaces looklike

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:25:b3:b3:25:80 brd ff:ff:ff:ff:ff:ff
3: ens1f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:25:b3:b3:25:81 brd ff:ff:ff:ff:ff:ff
4: enp2s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 10:1f:74:32:5d:70 brd ff:ff:ff:ff:ff:ff
    inet XXX.YYY.47.131/24 brd 130.166.47.255 scope global dynamic enp2s0f0
       valid_lft 21598sec preferred_lft 21598sec
    inet6 fe80::121f:74ff:fe32:5d70/64 scope link
       valid_lft forever preferred_lft forever
5: ens1f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:25:b3:b3:25:82 brd ff:ff:ff:ff:ff:ff
6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:25:b3:b3:25:83 brd ff:ff:ff:ff:ff:ff
7: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 10:1f:74:32:5d:74 brd ff:ff:ff:ff:ff:ff
    inet 10.47.1.131/16 brd 10.47.255.255 scope global dynamic enp2s0f1
       valid_lft 21598sec preferred_lft 21598sec
    inet6 fe80::121f:74ff:fe32:5d74/64 scope link
       valid_lft forever preferred_lft forever
8: ibs2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc fq_codel state UP group default qlen 256
    link/infiniband 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:4f:40:fd brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet6 fe80::202:c903:4f:40fd/64 scope link
       valid_lft forever preferred_lft forever
9: ibs2d1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc fq_codel state DOWN group default qlen 256
    link/infiniband 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:4f:40:fe brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

root@hph7a:~# networkctl
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 eth0 ether off pending
  3 ens1f1 ether off unmanaged
  4 enp2s0f0 ether routable configured
  5 ens1f0 ether off unmanaged
  6 eth4 ether off pending
  7 enp2s0f1 ether routable configured
  8 ibs2 infiniband degraded configuring
  9 ibs2d1 infiniband no-carrier configuring

9 links listed.

It will just stay in the degraded/configuring mode forever. iptables logging all UDP packets for ports 67/68 show no DHCP traffic at all.

But if I do dhclient ibs2 it becomes...
root@hph7a:~# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:25:b3:b3:25:80 brd ff:ff:ff:ff:ff:ff
3: ens1f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:25:b3:b3:25:81 brd ff:ff:ff:ff:ff:ff
4: enp2s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 10:1f:74:32:5d:70 brd ff:ff:ff:ff:ff:ff
    inet XXX.YYY.47.131/24 brd 130.166.47.255 scope global dynamic enp2s0f0
       valid_lft 21361sec preferred_lft 21361sec
    inet6 fe80::121f:74ff:fe32:5d70/64 scope link
       valid_lft forever preferred_lft forever
5: ens1f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:25:b3:b3:25:82 brd ff:ff:ff:ff:ff:ff
6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:25:b3:b3:25:83 brd ff:ff:ff:ff:ff:ff
7: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 10:1f:74:32:5d:74 brd ff:ff:ff:ff:ff:ff
    inet 10.47.1.131/16 brd 10.47.255.255 scope global dynamic enp2s0f1
       valid_lft 21361sec preferred_lft 21361sec
    inet6 fe80::121f:74ff:fe32:5d74/64 scope link
       valid_lft forever preferred_lft forever
8: ibs2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc fq_codel state UP group default qlen 256
    link/infiniband 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:4f:40:fd brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 192.168.47.131/24 brd 192.168.47.255 scope global dynamic ibs2
       valid_lft 21599sec preferred_lft 21599sec
    inet6 fe80::202:c903:4f:40fd/64 scope link
       valid_lft forever preferred_lft forever
9: ibs2d1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc fq_codel state DOWN group default qlen 256
    link/infiniband 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:4f:40:fe brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

So. My DHCP server works fine. manually calling a dhclient on the infiniband IPoIB interface works just fine. But netplan/networkd doesn't properly invoke DHCP clients to configure the interface.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: netplan (not installed)
ProcVersionSignature: Ubuntu 5.4.0-14.17-generic 5.4.18
Uname: Linux 5.4.0-14-generic x86_64
ApportVersion: 2.20.11-0ubuntu16
Architecture: amd64
Date: Sun Feb 16 16:07:06 2020
InstallationDate: Installed on 2020-02-16 (0 days ago)
InstallationMedia: Ubuntu-Server 19.10 "Eoan Ermine" - Release amd64 (20191017)
ProcEnviron:
 SHELL=/bin/bash
 LANG=en_US.UTF-8
 TERM=xterm
 PATH=(custom, no user)
SourcePackage: plan
UpgradeStatus: Upgraded to focal on 2020-02-16 (0 days ago)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Alexander Cramb (ac101m)]

I also have this issue.

Ubuntu, kernel:
20.04, 5.4.105-generic

Netplan configuration:
network:
  ethernets:
    ibp73s0:
      dhcp4: true
  renderer: NetworkManager
  version: 2

card:
Mellanox connectx-3

Adapter info (lspci -v):
49:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
 Subsystem: Mellanox Technologies MT27500 Family [ConnectX-3]
 Flags: bus master, fast devsel, latency 0, IRQ 116, NUMA node 0
 Memory at e7500000 (64-bit, non-prefetchable) [size=1M]
 Memory at e6000000 (64-bit, prefetchable) [size=8M]
 Expansion ROM at e7400000 [disabled] [size=1M]
 Capabilities: [40] Power Management version 3
 Capabilities: [48] Vital Product Data
 Capabilities: [9c] MSI-X: Enable+ Count=128 Masked-
 Capabilities: [60] Express Endpoint, MSI 00
 Capabilities: [c0] Vendor Specific Information: Len=18 <?>
 Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
 Capabilities: [148] Device Serial Number f4-52-14-03-00-2a-c1-50
 Capabilities: [154] Advanced Error Reporting
 Capabilities: [18c] Secondary PCI Express
 Kernel driver in use: mlx4_core
 Kernel modules: mlx4_core

Mellanox OFED version:
5.0-2.1.8.0-ubuntu20.04-x86_64

Network manager configuration:
[main]
plugins=ifupdown,keyfile
dns=default

[ifupdown]
managed=false

[device]
wifi.scan-rand-mac-address=no

[keyfile]
unmanaged-devices=interface-name:ibp73s0

Running netplan apply does not configure the interface. Running 'dhclient ibp73s0' does. Running 'netplan apply' after doing that causes the interface to lose it's IP and return to an unconfigured state.

I have a whole bunch of machines that are due to be upgraded from 16.04 before it goes end of life and they all have inifiniband interfaces configured with dhcp via /etc/network/interfaces (which works fine).