agent cannot be up on LXD/Fan network on OpenStack OVN/geneve mtu=1442
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Canonical Juju |
Fix Released
|
Medium
|
Joseph Phillips | ||
Bug Description
When one runs OpenStack with MTU=1500 underlying network, Neutron/OVN by default will create a tenant/overlay network with MTU=1442(1500-58). However, when deploying a workload on top, the Fan network with Juju will use MTU=1450 which is actually bigger than that. Then, Juju agent cannot be up inside LXD/Fan on top of OpenStack.
OpenStack deployment is based on https:/
and k8s as a workload on top of OpenStack is: https:/
$ openstack network list
+------
| ID | Name | Subnets |
+------
| ba9880e3-
| e27b1cef-
+------
$ openstack network show internal
+------
| Field | Value |
+------
...
| mtu | 1442 |
| name | internal |
...
| provider:
| provider:
| provider:
...
+------
$ juju machines -m k8s-on-openstack
Machine State DNS Inst id Series AZ Message
0 started 192.168.151.75 0e96a5b1-
0/lxd/0 pending juju-b9bd8b-0-lxd-0 focal nova Container started
1 started 192.168.151.66 6bfa5d2e-
[openstack instance/VM provisioned by Juju - ens3: mtu=1442, fan-252: mtu=1450]
juju-b9bd8b-
1: lo: <LOOPBACK,
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,
link/ether fa:16:3e:48:85:85 brd ff:ff:ff:ff:ff:ff
3: fan-252: <BROADCAST,
link/ether 06:28:c7:b4:50:eb brd ff:ff:ff:ff:ff:ff
4: ftun0: <BROADCAST,
link/ether 92:d7:07:6e:b4:db brd ff:ff:ff:ff:ff:ff
5: lxdbr0: <NO-CARRIER,
link/ether 00:16:3e:03:0e:96 brd ff:ff:ff:ff:ff:ff
7: 0lxd0-0@if6: <BROADCAST,
link/ether 06:28:c7:b4:50:eb brd ff:ff:ff:ff:ff:ff link-netnsid 0
[lxc config of LXD container as juju machine 0/lxd/0]
juju-b9bd8b-
...
user.user-data: |
#cloud-config
apt_mirror: ""
bootcmd:
- install -D -m 644 /dev/null '/etc/netplan/
- |-
printf '%s\n' 'network:
version: 2
ethernets:
eth0:
match:
dhcp4: true
mtu: 1450
...
devices:
eth0:
host_name: 0lxd0-0
hwaddr: 00:16:3e:82:f9:44
mtu: "1450"
name: eth0
nictype: bridged
parent: fan-252
type: nic
[cloud-
Cloud-init v. 21.2-3-
Hit:1 http://
Err:2 http://
Connection failed [IP: 192.168.151.1 8000]
Err:3 http://
Connection failed [IP: 192.168.151.1 8000]
Err:4 http://
Connection failed [IP: 192.168.151.1 8000]
Ign:5 http://
Ign:6 http://
Ign:7 http://
Ign:8 http://
Ign:9 http://
Ign:10 http://
Get:5 http://
Get:5 http://
Get:5 http://
Get:5 http://
Get:5 http://
Get:5 http://
+ printf Attempt 5 to download agent binaries from %s...\n https:/
Attempt 5 to download agent binaries from https:/
+ curl -sSfw agent binaries from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s --connect-timeout 20 --noproxy * --insecure -o /var/lib/
curl: (28) Operation timed out after 20000 milliseconds with 0 out of 0 bytes received
agent binaries from https:/
Download failed, retrying in 15s
+ sleep 15
[manually run tracepath to get pmtu]
juju-b9bd8b-
1?: [LOCALHOST] pmtu 1450
1: 252.16.0.1 0.346ms
1: 252.16.0.1 0.051ms
2: 252.16.0.1 0.053ms pmtu 1442
2: no reply
3: 192.168.151.1 3.283ms reached
Resume: pmtu 1442 hops 3 back 3
| Nobuto Murata (nobuto) wrote | #1 |
| Nobuto Murata (nobuto) wrote | #2 |
| Joseph Phillips (manadart) wrote | #3 |
| Changed in juju: | |
| status: | New → Triaged |
| importance: | Undecided → Medium |
| assignee: | nobody → Joseph Phillips (manadart) |
| Joseph Phillips (manadart) wrote | #4 |
| Changed in juju: | |
| status: | Triaged → In Progress |
| milestone: | none → 2.9.17 |
| Joseph Phillips (manadart) wrote | #5 |
| Nobuto Murata (nobuto) wrote | #6 |
| Nobuto Murata (nobuto) wrote | #7 |
| Changed in juju: | |
| milestone: | 2.9.17 → 2.9.18 |
| Joseph Phillips (manadart) wrote | #8 |
| no longer affects: | ubuntu-fan (Ubuntu) |
| Joseph Phillips (manadart) wrote | #9 |
| Changed in juju: | |
| status: | In Progress → Fix Committed |
| Changed in juju: | |
| status: | Fix Committed → Fix Released |
For the record, by lowering MTU of eth0 on LXD on top of OpenStack VM by hand as `ip link set eth0 mtu 1442`, that makes binary download successful.
+ printf Attempt 17 to download agent binaries from %s...\n https:/
Attempt 17 to download agent binaries from https:/
+ curl -sSfw agent binaries from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s --connect-timeout 20 --noproxy * --insecure
-o /var/lib/
curl: (28) Operation timed out after 20001 milliseconds with 0 out of 0 bytes received
agent binaries from https:/
failed, retrying in 15s
Download failed, retrying in 15s
+ sleep 15
+ n=18
+ true
+ printf Attempt 18 to download agent binaries from %s...\n https:/
Attempt 18 to download agent binaries from https:/
+ curl -sSfw agent binaries from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s --connect-timeout 20 --noproxy * --insecure
-o /var/lib/
agent binaries from https:/
echo Agent binaries downloaded successfully.
Agent binaries downloaded successfully.
+ break
+ sha256sum /var/lib/
+ grep 635a3524acc09a5
635a3524acc09a5