For the first part there is an elegant solution: force ongoing connections to initiate from the static ip of a controller. This behavior is ensured when preferred_lft 0 is set on an ip address of an interface (as described here: http://www.davidc.net/networking/ipv6-source-address-selection-linux).
This would allow us to assign a floating IP and a static ip from the same subnet to the management network but at the same time make sure that all outgoing connections to ip's on other hosts in this subnet are initiated from the static ip.
In the example below we have the active controller with 3 ip addresses on the management interface, one static and two floating. The floating IPs move from controller to controller on swact:
1. static: fd01::2/64
2. floating: fd01::1/64
3. floating: fd01::4/64
Connections w/o the option:
====================
[And give example here]
We can see that there are connections opened from the floating IPs.
Connection with the option:
====================
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp6 0 0 :::6800 :::* LISTEN 2587586/ceph-osd
tcp6 0 0 :::6801 :::* LISTEN 2587586/ceph-osd
tcp6 0 0 :::6802 :::* LISTEN 2587586/ceph-osd
tcp6 0 0 :::6803 :::* LISTEN 2587586/ceph-osd
tcp6 0 0 fd01::2:6789 :::* LISTEN 2587301/ceph-mon
tcp6 0 0 fd01::2:40924 fd01::3:6789 ESTABLISHED 2587586/ceph-osd
tcp6 0 0 fd01::2:40888 fd01::3:6789 ESTABLISHED 2587301/ceph-mon
tcp6 0 0 fd01::2:45520 fd01::3:6800 ESTABLISHED 2587586/ceph-osd
tcp6 0 0 fd01::2:43872 fd01::3:6804 ESTABLISHED 2587586/ceph-osd
tcp6 0 0 fd01::2:6801 fd01::3:47520 ESTABLISHED 2587586/ceph-osd
tcp6 0 0 fd01::2:6802 fd01::3:53458 ESTABLISHED 2587586/ceph-osd
tcp6 0 0 fd01::2:6803 fd01::3:48860 ESTABLISHED 2587586/ceph-osd
tcp6 0 0 fd01::2:48158 fd01::15f6:1bc9:22:6789 ESTABLISHED 2587301/ceph-mon
tcp6 0 0 fd01::2:58020 fd01::3:6803 ESTABLISHED 2587586/ceph-osd
tcp6 0 0 fd01::2:34342 fd01::3:6800 ESTABLISHED 81962/ceph-mgr
tcp6 0 0 fd01::2:45484 fd01::3:6800 ESTABLISHED 2587301/ceph-mon
tcp6 0 0 fd01::2:48000 fd01::15f6:1bc9:22:6789 ESTABLISHED 81962/ceph-mgr
How to configure the option by hand
===================
Result:
6: vlan10@enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000
link/ether 08:00:27:66:a6:e8 brd ff:ff:ff:ff:ff:ff
inet6 fd01::1/64 scope global deprecated
valid_lft forever preferred_lft 0sec
inet6 fd01::4/64 scope global deprecated
valid_lft forever preferred_lft 0sec
inet6 fd01::2/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fe66:a6e8/64 scope link
valid_lft forever preferred_lft forever
For the first part there is an elegant solution: force ongoing connections to initiate from the static ip of a controller. This behavior is ensured when preferred_lft 0 is set on an ip address of an interface (as described here: http:// www.davidc. net/networking/ ipv6-source- address- selection- linux).
This would allow us to assign a floating IP and a static ip from the same subnet to the management network but at the same time make sure that all outgoing connections to ip's on other hosts in this subnet are initiated from the static ip.
In the example below we have the active controller with 3 ip addresses on the management interface, one static and two floating. The floating IPs move from controller to controller on swact:
1. static: fd01::2/64
2. floating: fd01::1/64
3. floating: fd01::4/64
Connections w/o the option: ======= ======
=======
[And give example here]
We can see that there are connections opened from the floating IPs.
Connection with the option: ======= ====== 1bc9:22: 6789 ESTABLISHED 2587301/ceph-mon 1bc9:22: 6789 ESTABLISHED 81962/ceph-mgr
=======
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp6 0 0 :::6800 :::* LISTEN 2587586/ceph-osd
tcp6 0 0 :::6801 :::* LISTEN 2587586/ceph-osd
tcp6 0 0 :::6802 :::* LISTEN 2587586/ceph-osd
tcp6 0 0 :::6803 :::* LISTEN 2587586/ceph-osd
tcp6 0 0 fd01::2:6789 :::* LISTEN 2587301/ceph-mon
tcp6 0 0 fd01::2:40924 fd01::3:6789 ESTABLISHED 2587586/ceph-osd
tcp6 0 0 fd01::2:40888 fd01::3:6789 ESTABLISHED 2587301/ceph-mon
tcp6 0 0 fd01::2:45520 fd01::3:6800 ESTABLISHED 2587586/ceph-osd
tcp6 0 0 fd01::2:43872 fd01::3:6804 ESTABLISHED 2587586/ceph-osd
tcp6 0 0 fd01::2:6801 fd01::3:47520 ESTABLISHED 2587586/ceph-osd
tcp6 0 0 fd01::2:6802 fd01::3:53458 ESTABLISHED 2587586/ceph-osd
tcp6 0 0 fd01::2:6803 fd01::3:48860 ESTABLISHED 2587586/ceph-osd
tcp6 0 0 fd01::2:48158 fd01::15f6:
tcp6 0 0 fd01::2:58020 fd01::3:6803 ESTABLISHED 2587586/ceph-osd
tcp6 0 0 fd01::2:34342 fd01::3:6800 ESTABLISHED 81962/ceph-mgr
tcp6 0 0 fd01::2:45484 fd01::3:6800 ESTABLISHED 2587301/ceph-mon
tcp6 0 0 fd01::2:48000 fd01::15f6:
How to configure the option by hand
===================
Result: MULTICAST, UP,LOWER_ UP> mtu 1500 qdisc htb state UP group default qlen 1000 27ff:fe66: a6e8/64 scope link
6: vlan10@enp0s8: <BROADCAST,
link/ether 08:00:27:66:a6:e8 brd ff:ff:ff:ff:ff:ff
inet6 fd01::1/64 scope global deprecated
valid_lft forever preferred_lft 0sec
inet6 fd01::4/64 scope global deprecated
valid_lft forever preferred_lft 0sec
inet6 fd01::2/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::a00:
valid_lft forever preferred_lft forever