| 2024-02-01 11:53:13 |
Jan Wasilewski |
description |
[High-Level Description]
While creating an amphora from the Octavia service, we are encountering a race condition. Nova-compute is unable to add a QC ingress rule because the tap interface does not exist yet. This situation results in the service becoming temporarily unavailable, failing in approximately 50% of cases.
[Pre-conditions]
Create a simple octavia loadbalancer with environment deployed by kolla-environment.
[Step-by-Step Reproduction]
Create a load balancer as part of a Heat stack, following the example provided at: https://github.com/syseleven/heat-examples/tree/master/lbaas-octavia-http
[Expected Output]
Expect 100% successful creation of the amphora and subsequent Nova instances serving as members for the load balancer.
[Actual Output]
During the creation process, it is observed that nova-compute fails due to:
/var/log/kolla/libvirt/libvirtd.log:2024-01-31 09:24:55.852+0000: 4065628: error : virCommandWait:2748 : internal error: Child process (tc filter add dev tap5f365c23-89 parent ffff: protocol all u32 match u32 0 0 police rate 64000kbps burst 64000kb mtu 64kb drop flowid :1) unexpected exit status 2: Error: Parent Qdisc doesn't exists.
/var/log/kolla/nova/nova-compute.log:2024-01-31 10:24:56.086 7 ERROR nova.virt.libvirt.guest libvirt.libvirtError: internal error: Child process (tc filter add dev tap5f365c23-89 parent ffff: protocol all u32 match u32 0 0 police rate 64000kbps burst 64000kb mtu 64kb drop flowid :1) unexpected exit status 2: Error: Parent Qdisc doesn't exists.
Unfortunately ovs-vswitchd.log file says that tap interface was not created at this point:
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.469Z|00094|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.809Z|00095|bridge|INFO|bridge br-int: added interface tap5f365c23-89 on port 27
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.885Z|00096|bridge|INFO|bridge br-int: deleted interface tap5f365c23-89 on port 27
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.888Z|00097|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.927Z|00098|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.934Z|00099|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.943Z|00100|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:56.024Z|00101|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
[Version]
OpenStack Zed, deployed by kolla-ansible with all defaults
Ubuntu 22.04 LTS
libvirt: 8.0.0-1ubuntu7.7
nova: zed
python3-openvswitch: 3.0.3-0ubuntu0.22.10.3~cloud3
neutron: zed |
[High-Level Description]
While creating an amphora from the Octavia service, we are encountering a race condition. Nova-compute is unable to add a QC ingress rule because the tap interface does not exist yet. This situation results in the service becoming temporarily unavailable, failing in approximately 50% of cases.
[Pre-conditions]
Create a simple octavia loadbalancer with environment deployed by kolla-environment. For amphora server, use flavor with qos properties as from example: https://paste.openstack.org/show/bkx3RgEFqUwxCU0Qr8z2/
[Step-by-Step Reproduction]
Create a load balancer as part of a Heat stack, following the example provided at: https://github.com/syseleven/heat-examples/tree/master/lbaas-octavia-http
Use a qos flavor as shown in example: https://paste.openstack.org/show/bkx3RgEFqUwxCU0Qr8z2/
[Expected Output]
Expect 100% successful creation of the amphora and subsequent Nova instances serving as members for the load balancer.
[Actual Output]
During the creation process, it is observed that nova-compute fails due to:
/var/log/kolla/libvirt/libvirtd.log:2024-01-31 09:24:55.852+0000: 4065628: error : virCommandWait:2748 : internal error: Child process (tc filter add dev tap5f365c23-89 parent ffff: protocol all u32 match u32 0 0 police rate 64000kbps burst 64000kb mtu 64kb drop flowid :1) unexpected exit status 2: Error: Parent Qdisc doesn't exists.
/var/log/kolla/nova/nova-compute.log:2024-01-31 10:24:56.086 7 ERROR nova.virt.libvirt.guest libvirt.libvirtError: internal error: Child process (tc filter add dev tap5f365c23-89 parent ffff: protocol all u32 match u32 0 0 police rate 64000kbps burst 64000kb mtu 64kb drop flowid :1) unexpected exit status 2: Error: Parent Qdisc doesn't exists.
Unfortunately ovs-vswitchd.log file says that tap interface was not created at this point:
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.469Z|00094|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.809Z|00095|bridge|INFO|bridge br-int: added interface tap5f365c23-89 on port 27
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.885Z|00096|bridge|INFO|bridge br-int: deleted interface tap5f365c23-89 on port 27
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.888Z|00097|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.927Z|00098|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.934Z|00099|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.943Z|00100|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
/var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:56.024Z|00101|bridge|WARN|could not open network device tap5f365c23-89 (No such device)
[Version]
OpenStack Zed, deployed by kolla-ansible with all defaults
Ubuntu 22.04 LTS
libvirt: 8.0.0-1ubuntu7.7
nova: zed
python3-openvswitch: 3.0.3-0ubuntu0.22.10.3~cloud3
neutron: zed |
|
