Jammy Charmed OpenStack deployment fails over connectivity issues when using converged OVS bridge for control and data planes
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Confirmed
|
Undecided
|
Unassigned | ||
Jammy |
Fix Released
|
Medium
|
Unassigned |
Bug Description
Platform: OpenStack Yoga, Ubuntu 22.04 Jammy, Kernel 5.15.0-37-generic
Charmed Openstack deployment with HW Offload over Jammy series will look ok until Vault initialization phase, then after initializing Vault all DB-related apps will end up in block/error state over "Failed to connect to MYSQL".
Connectivity testing between DB containers located on different nodes will show there is unexplained sporadic packet loss preventing proper communication between the DB related apps.
This will happen when the following conditions are met:
1. Control plane (oam, internal spaces) is configured as vlan interfaces on the same OVS bridge used for data plane (over high speed NIC with HW Offload capabilities).
2. OVS was set with HW offload=true (will happen by OVN chrams after Vault initialization)
3. NIC was not yet set to "switchdev" mode (netplan file will be created by OVN chrams after Vault initialization, however will take affect only after node is rebooted)
The root cause is the following missing kernel patch:
https:/
To reproduce:
Deploy charmed openstack with HW offload while using control plane on the high speed NIC OVS bridge. Before initializing Vault login to one of the innoDB instances and ping the other 2 instances - all ok. Manually enable OVS HW Offload, ping will become inconsistent.
Workaround:
After the deployment bring-up phase, BEFORE enabling Vault, login to the nodes and manually create 150-charm-ovn.yaml (example below). Then reboot one node after another. When nodes recover proceed with Vault initialization to complete the deployment.
#root@node3:
#######
# [ WARNING ]
# Configuration file maintained by Juju. Local changes may be overwritten.
# Config managed by ovn-chassis charm
#######
network:
version: 2
ethernets:
ens1f0:
virtual-
embedded-
delay-
ens1f1:
virtual-
embedded-
delay-
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Jul 4 10:46 seq
crw-rw---- 1 root audio 116, 33 Jul 4 10:46 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu82.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
CRDA: N/A
CasperMD5CheckR
DistroRelease: Ubuntu 22.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: HP ProLiant DL360 Gen9
NonfreeKernelMo
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=xterm-
PATH=(custom, no user)
LANG=C.UTF-8
SHELL=/bin/bash
ProcFB: 0 mgag200drmfb
ProcKernelCmdLine: BOOT_IMAGE=
ProcVersionSign
RelatedPackageV
linux-
linux-
linux-firmware 20220329.
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: jammy uec-images
Uname: Linux 5.15.0-40-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 03/05/2015
dmi.bios.release: 1.32
dmi.bios.vendor: HP
dmi.bios.version: P89
dmi.chassis.type: 23
dmi.chassis.vendor: HP
dmi.ec.
dmi.modalias: dmi:bvnHP:
dmi.product.family: ProLiant
dmi.product.name: ProLiant DL360 Gen9
dmi.product.sku: 755258-B21
dmi.sys.vendor: HP
CVE References
no longer affects: | plan (Ubuntu) |
Changed in linux (Ubuntu Jammy): | |
importance: | Undecided → Medium |
status: | New → In Progress |
Changed in linux (Ubuntu Jammy): | |
status: | In Progress → Fix Committed |
tags: |
added: verification-done-jammy removed: apport-collected jammy uec-images verification-needed-jammy |
apport information