Charms (not workload) max out CPUs and network bandwidth degrades

[Nobuto Murata]

$ snap list openstack
Name Version Rev Tracking Publisher Notes
openstack 2024.1 569 2024.1/edge canonical✓ -

I noticed inconsistent network bandwidth in pod-to-pod communication.

This is iperf3 between keystone-0 and keystone-1. As you can see the bandwidth is ~2Gbps in the best case and 50-500Mbps sometimes. By looking at the CPU usage, charm hooks are saturating the CPU when the bandwidth gets degraded.

Similar to:
https://bugs.launchpad.net/sunbeam-charms/+bug/2066205
(this was a pebble health check)

# iperf3 -c 10.1.51.46 -t 60
Connecting to host 10.1.51.46, port 5201
[ 5] local 10.1.48.119 port 59348 connected to 10.1.51.46 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 6.74 MBytes 56.5 Mbits/sec 28 212 KBytes
[ 5] 1.00-2.00 sec 12.9 MBytes 108 Mbits/sec 36 246 KBytes
[ 5] 2.00-3.00 sec 14.0 MBytes 118 Mbits/sec 0 284 KBytes
[ 5] 3.00-4.01 sec 16.9 MBytes 141 Mbits/sec 10 325 KBytes
[ 5] 4.01-5.00 sec 11.1 MBytes 93.9 Mbits/sec 0 348 KBytes
[ 5] 5.00-6.00 sec 15.4 MBytes 129 Mbits/sec 3 375 KBytes
[ 5] 6.00-7.00 sec 19.3 MBytes 162 Mbits/sec 17 410 KBytes
[ 5] 7.00-8.00 sec 70.4 MBytes 590 Mbits/sec 2 515 KBytes
[ 5] 8.00-9.00 sec 184 MBytes 1.54 Gbits/sec 4 710 KBytes
[ 5] 9.00-10.00 sec 218 MBytes 1.82 Gbits/sec 153 848 KBytes
[ 5] 10.00-11.00 sec 209 MBytes 1.75 Gbits/sec 107 978 KBytes
[ 5] 11.00-12.00 sec 226 MBytes 1.90 Gbits/sec 0 1.03 MBytes
[ 5] 12.00-13.00 sec 202 MBytes 1.70 Gbits/sec 17 1.07 MBytes
[ 5] 13.00-14.00 sec 191 MBytes 1.60 Gbits/sec 13 1.15 MBytes
[ 5] 14.00-15.00 sec 204 MBytes 1.71 Gbits/sec 7 1.19 MBytes
[ 5] 15.00-16.00 sec 226 MBytes 1.90 Gbits/sec 0 1.21 MBytes

...

[ 5] 48.00-49.00 sec 232 MBytes 1.95 Gbits/sec 3 1.48 MBytes
[ 5] 49.00-50.00 sec 214 MBytes 1.79 Gbits/sec 0 1.57 MBytes
[ 5] 50.00-51.00 sec 216 MBytes 1.81 Gbits/sec 1 1.63 MBytes
[ 5] 51.00-52.01 sec 120 MBytes 994 Mbits/sec 18 1.64 MBytes
[ 5] 52.01-53.00 sec 32.5 MBytes 276 Mbits/sec 9 1.65 MBytes
[ 5] 53.00-54.00 sec 38.8 MBytes 325 Mbits/sec 55 1.20 MBytes
[ 5] 54.00-55.00 sec 63.8 MBytes 534 Mbits/sec 57 1.23 MBytes
[ 5] 55.00-56.00 sec 48.8 MBytes 409 Mbits/sec 2 1.25 MBytes
[ 5] 56.00-57.03 sec 23.8 MBytes 194 Mbits/sec 38 1.26 MBytes
[ 5] 57.03-58.00 sec 32.5 MBytes 280 Mbits/sec 0 1.28 MBytes
[ 5] 58.00-59.02 sec 46.2 MBytes 381 Mbits/sec 30 1.30 MBytes
[ 5] 59.02-60.00 sec 36.2 MBytes 310 Mbits/sec 0 1.32 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.00 sec 9.64 GBytes 1.38 Gbits/sec 2108 sender
[ 5] 0.00-60.07 sec 9.64 GBytes 1.38 Gbits/sec receiver

iperf Done.