[High-Level Description]
While creating an amphora from the Octavia service, we are encountering a race condition. Nova-compute is unable to add a QC ingress rule because the tap interface does not exist yet. This situation results in the service becoming temporarily unavailable, failing in approximately 50% of cases.
[Pre-conditions]
Create a simple octavia loadbalancer with environment deployed by kolla-environment.
[Expected Output]
Expect 100% successful creation of the amphora and subsequent Nova instances serving as members for the load balancer.
[Actual Output]
During the creation process, it is observed that nova-compute fails due to:
/var/log/kolla/libvirt/libvirtd.log:2024-01-31 09:24:55.852+0000: 4065628: error : virCommandWait:2748 : internal error: Child process (tc filter add dev tap5f365c23-89 parent ffff: protocol all u32 match u32 0 0 police rate 64000kbps burst 64000kb mtu 64kb drop flowid :1) unexpected exit status 2: Error: Parent Qdisc doesn't exists.
/var/log/kolla/nova/nova-compute.log:2024-01-31 10:24:56.086 7 ERROR nova.virt.libvirt.guest libvirt.libvirtError: internal error: Child process (tc filter add dev tap5f365c23-89 parent ffff: protocol all u32 match u32 0 0 police rate 64000kbps burst 64000kb mtu 64kb drop flowid :1) unexpected exit status 2: Error: Parent Qdisc doesn't exists.
Unfortunately ovs-vswitchd.log file says that tap interface was not created at this point:
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.469Z|00094|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.809Z|00095|bridge|INFO|bridge br-int: added interface tap5f365c23-89 on port 27
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.885Z|00096|bridge|INFO|bridge br-int: deleted interface tap5f365c23-89 on port 27
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.888Z|00097|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.927Z|00098|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.934Z|00099|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.943Z|00100|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:56.024Z|00101|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
[Version]
OpenStack Zed, deployed by kolla-ansible with all defaults
Ubuntu 22.04 LTS
libvirt: 8.0.0-1ubuntu7.7
nova: zed
python3-openvswitch: 3.0.3-0ubuntu0.22.10.3~cloud3
neutron: zed
[High-Level Description]
While creating an amphora from the Octavia service, we are encountering a race condition. Nova-compute is unable to add a QC ingress rule because the tap interface does not exist yet. This situation results in the service becoming temporarily unavailable, failing in approximately 50% of cases.
[Pre-conditions]
Create a simple octavia loadbalancer with environment deployed by kolla-environment.
[Step-by-Step Reproduction] /github. com/syseleven/ heat-examples/ tree/master/ lbaas-octavia- http
Create a load balancer as part of a Heat stack, following the example provided at: https:/
[Expected Output]
Expect 100% successful creation of the amphora and subsequent Nova instances serving as members for the load balancer.
[Actual Output] kolla/libvirt/ libvirtd. log:2024- 01-31 09:24:55.852+0000: 4065628: error : virCommandWait:2748 : internal error: Child process (tc filter add dev tap5f365c23-89 parent ffff: protocol all u32 match u32 0 0 police rate 64000kbps burst 64000kb mtu 64kb drop flowid :1) unexpected exit status 2: Error: Parent Qdisc doesn't exists. kolla/nova/ nova-compute. log:2024- 01-31 10:24:56.086 7 ERROR nova.virt. libvirt. guest libvirt. libvirtError: internal error: Child process (tc filter add dev tap5f365c23-89 parent ffff: protocol all u32 match u32 0 0 police rate 64000kbps burst 64000kb mtu 64kb drop flowid :1) unexpected exit status 2: Error: Parent Qdisc doesn't exists.
During the creation process, it is observed that nova-compute fails due to:
/var/log/
/var/log/
Unfortunately ovs-vswitchd.log file says that tap interface was not created at this point: kolla/openvswit ch/ovs- vswitchd. log:2024- 01-31T09: 24:55.469Z| 00094|bridge| WARN|could not open network device tap5f365c23-89 (No such device) kolla/openvswit ch/ovs- vswitchd. log:2024- 01-31T09: 24:55.809Z| 00095|bridge| INFO|bridge br-int: added interface tap5f365c23-89 on port 27 kolla/openvswit ch/ovs- vswitchd. log:2024- 01-31T09: 24:55.885Z| 00096|bridge| INFO|bridge br-int: deleted interface tap5f365c23-89 on port 27 kolla/openvswit ch/ovs- vswitchd. log:2024- 01-31T09: 24:55.888Z| 00097|bridge| WARN|could not open network device tap5f365c23-89 (No such device) kolla/openvswit ch/ovs- vswitchd. log:2024- 01-31T09: 24:55.927Z| 00098|bridge| WARN|could not open network device tap5f365c23-89 (No such device) kolla/openvswit ch/ovs- vswitchd. log:2024- 01-31T09: 24:55.934Z| 00099|bridge| WARN|could not open network device tap5f365c23-89 (No such device) kolla/openvswit ch/ovs- vswitchd. log:2024- 01-31T09: 24:55.943Z| 00100|bridge| WARN|could not open network device tap5f365c23-89 (No such device) kolla/openvswit ch/ovs- vswitchd. log:2024- 01-31T09: 24:56.024Z| 00101|bridge| WARN|could not open network device tap5f365c23-89 (No such device)
/var/log/
/var/log/
/var/log/
/var/log/
/var/log/
/var/log/
/var/log/
/var/log/
[Version] openvswitch: 3.0.3-0ubuntu0. 22.10.3~ cloud3
OpenStack Zed, deployed by kolla-ansible with all defaults
Ubuntu 22.04 LTS
libvirt: 8.0.0-1ubuntu7.7
nova: zed
python3-
neutron: zed