Reboot of Amphora does not work with altname in ip link

Bug #2027995 reported by Daniel Failing
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
octavia
Confirmed
Medium
Unassigned

Bug Description

In newer kernel/ip versions there are aliases for the interfaces configured.
Because of this, the interfaces for the member and listener are not moved to the correct network namespace.

Line in question:
https://opendev.org/openstack/octavia/src/branch/master/octavia/amphorae/backends/agent/api_server/templates/amphora-netns.systemd.j2#L29

Output from /sbin/ip link:
```
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether fa:aa:bb:cc:11:21 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
3: ens7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether fa:aa:bb:cc:11:22 brd ff:ff:ff:ff:ff:ff
    altname enp0s7
```
With this `awk '{getline n; print $0,n}'` will contatinate the wrong lines.
It can be fixed by adding `grep -e mtu -e link`, but maybe a better approach for this line is needed.

Revision history for this message
Gregory Thiemonge (gthiemonge) wrote :

I confirm this issue, after rebooting a centos 9 stream amphora, the vip and member interfaces are in the "main" network namespace:

bash-5.1# cat /var/lib/octavia/plugged_interfaces
fa:16:3e:34:0c:9e eth1
fa:16:3e:33:a4:5d eth2
bash-5.1# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:e6:32:a8 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    inet 192.168.0.172/24 brd 192.168.0.255 scope global noprefixroute ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fee6:32a8/64 scope link
       valid_lft forever preferred_lft forever
3: ens7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:34:0c:9e brd ff:ff:ff:ff:ff:ff
    altname enp0s7
4: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:33:a4:5d brd ff:ff:ff:ff:ff:ff
    altname enp0s8

the amphora-interface application cannot configure those interfaces:

Jul 17 11:43:32 amphora-5a7dab8e-fbeb-4c15-be6b-b0e698dffe8b.novalocal ip[475]: 2023-07-17 11:43:32.815 475 INFO octavia.common.config [-] Logging enabled!
Jul 17 11:43:32 amphora-5a7dab8e-fbeb-4c15-be6b-b0e698dffe8b.novalocal ip[475]: 2023-07-17 11:43:32.817 475 INFO octavia.common.config [-] /usr/local/bin/amphora-interface version 12.1.0.dev72
Jul 17 11:43:32 amphora-5a7dab8e-fbeb-4c15-be6b-b0e698dffe8b.novalocal ip[475]: 2023-07-17 11:43:32.822 475 INFO octavia.amphorae.backends.utils.interface [-] Setting interface eth2 up
Jul 17 11:43:32 amphora-5a7dab8e-fbeb-4c15-be6b-b0e698dffe8b.novalocal ip[475]: 2023-07-17 11:43:32.824 475 INFO octavia.amphorae.backends.utils.interface [-] Setting interface eth1 up
Jul 17 11:43:32 amphora-5a7dab8e-fbeb-4c15-be6b-b0e698dffe8b.novalocal ip[475]: 2023-07-17 11:43:32.825 475 INFO octavia.amphorae.backends.utils.interface [-] Setting interface lo up
Jul 17 11:43:32 amphora-5a7dab8e-fbeb-4c15-be6b-b0e698dffe8b.novalocal ip[475]: Error: Could not configure interface: Error on action 'up' for interface eth2: list index out of range., Error on action 'up' for interface eth1: list index out of range.

Changed in octavia:
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
Thong Tran (thongth98) wrote :

I encountered an issue related to network interface name alias support in the new kernel, specifically while testing on Ubuntu 22.04.

To resolve this problem, I made adjustments to a script responsible for assigning interfaces to namespaces. These adjustments involved removing the alternate name using the 'grep -v "altname"' command.

Link source:
https://opendev.org/openstack/octavia/src/branch/master/octavia/amphorae/backends/agent/api_server/templates/amphora-netns.systemd.j2#L29

Fix:
ExecStart=-/bin/sh -c '/sbin/ip link | grep -v "altname" | awk \'{getline n; print $0,n}\' | awk \'{sub(":","",$2)} { for(i=1;i<=NF;i++) if ($i == "link/ether") {print $(i+1) " " $2} }\' | sort -k 1 | join -j 1 - /var/lib/octavia/plugged_interfaces.sorted | awk \'{system("ip link set "$2" netns amphora-haproxy name "$3"")}\''

Revision history for this message
Gregory Thiemonge (gthiemonge) wrote :

Thanks for the fixes, I think this line is unreadable and I would be in favor of:
- fixing this issue with a grep (-v altname or -e mtu -e link) on master and stable branches
- creating a readable (python?) script that would replace some of the messy things that are in this systemd file (probably not backportable)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.