Hello, we've narrowed down the problem to the upgrade of the python3-openvswitch package out of the 42 total python packages that were changed during the routine upgrade. During the upgrade it went from 2.13.3-0ubuntu0.20.04.2 -> 2.13.8-0ubuntu1.1 In the release history of the openvswitch package of which python3-openvswitch is a dependency and is versioned together, 2.13.5-0ubuntu1 is the first version available after the package we had in the Octavia containers before the upgrade. It presents the problematic behavior and in the latest available version 2.13.8-0ubuntu1.2, it is still there. We were only specifically testing what package change was causing Octavia API to hang (not the full loadbalancer functionality) as we've installed one python package deb at a time without dependency through dpdk. This behavior can be replicated by having at least one loadbalancer object created in the Octavia DB and running 2-3 openstack loadbalancer list watch screens to query the 4 API workers under apache wsgi while reloading the apache workers every 60 seconds to stress as this does not happen every time, but it should be visible behavior under these conditions after 1-2 hours. deb packages taken from: https://launchpad.net/ubuntu/focal/amd64/python3-openvswitch/2.13.3-0ubuntu0.20.04.2 https://launchpad.net/ubuntu/focal/amd64/python3-openvswitch/2.13.5-0ubuntu1 This behavior also does not happen if the Octavia OVN provider is disabled with all the latest packages but with the python3-openvswitch downgrade it does not appear even with the OVN driver enabled. [api_settings] #enabled_provider_drivers = amphora:The Octavia Amphora driver,ovn:Octavia OVN driver enabled_provider_drivers = amphora:The Octavia Amphora driver # Python version and libs installed ubuntu@juju-7bae07-1-lxd-7:~$ dpkg -l | grep python3.8 ii libpython3.8:amd64 3.8.10-0ubuntu1~20.04.8 amd64 Shared Python runtime library (version 3.8) ii libpython3.8-dev:amd64 3.8.10-0ubuntu1~20.04.8 amd64 Header files and a static library for Python (v3.8) ii libpython3.8-minimal:amd64 3.8.10-0ubuntu1~20.04.8 amd64 Minimal subset of the Python language (version 3.8) ii libpython3.8-stdlib:amd64 3.8.10-0ubuntu1~20.04.8 amd64 Interactive high-level object-oriented language (standard library, version 3.8) ii python3.8 3.8.10-0ubuntu1~20.04.8 amd64 Interactive high-level object-oriented language (version 3.8) ii python3.8-dev 3.8.10-0ubuntu1~20.04.8 amd64 Header files and a static library for Python (v3.8) ii python3.8-minimal 3.8.10-0ubuntu1~20.04.8 amd64 Minimal subset of the Python language (version 3.8) # Octavia, ovn and openvswitch installed packages ubuntu@juju-7bae07-1-lxd-7:~$ dpkg -l | grep -e openvswitch -e ovn -e octavia ii octavia-api 6.2.2-0ubuntu1 all OpenStack Load Balancer as a Service - API frontend ii octavia-common 6.2.2-0ubuntu1 all OpenStack Load Balancer as a Service - Common files ii octavia-driver-agent 6.2.2-0ubuntu1 all OpenStack Load Balancer Service - Driver Agent ii octavia-health-manager 6.2.2-0ubuntu1 all OpenStack Load Balancer Service - Health manager ii octavia-housekeeping 6.2.2-0ubuntu1 all OpenStack Load Balancer Service - Housekeeping manager ii octavia-worker 6.2.2-0ubuntu1 all OpenStack Load Balancer Service - Worker ii openvswitch-common 2.13.8-0ubuntu1.2 amd64 Open vSwitch common components ii openvswitch-switch 2.13.8-0ubuntu1.2 amd64 Open vSwitch switch implementations ii ovn-common 20.03.2-0ubuntu0.20.04.4 amd64 OVN common components ii ovn-host 20.03.2-0ubuntu0.20.04.4 amd64 OVN host components ii python3-octavia 6.2.2-0ubuntu1 all OpenStack Load Balancer as a Service - Python libraries ii python3-octavia-lib 2.0.0-0ubuntu1 all Library to support Octavia provider drivers ii python3-openvswitch 2.13.8-0ubuntu1.2 all Python 3 bindings for Open vSwitch ii python3-ovn-octavia-provider 0.1.0-0ubuntu1 all OpenStack Octavia Integration with OVN - Python 3 library # All installed dependencies for ovn, octavia and openvswitch for pack in $(for p in $(dpkg -l | grep -e ovn -e openvswitch -e octavia | awk '{print $2}' | sort | uniq); do apt-cache depends $p; done | grep Depends | sort | uniq | awk '{print $2}'); do dpkg -l | grep $pack; done | sort | uniq ii adduser 3.118ubuntu2 all add and remove users and groups ii dbconfig-common 2.0.13 all framework that helps packages to manage databases ii debconf 1.5.73 all Debian configuration management system ii debconf-i18n 1.5.73 all full internationalization support for debconf ii init-system-helpers 1.57 all helper tools for all init systems ii kmod 27-1ubuntu2.1 amd64 tools for managing Linux kernel modules ii libc6-dev:amd64 2.31-0ubuntu9.9 amd64 GNU C Library: Development Libraries and Header Files ii libc6:amd64 2.31-0ubuntu9.9 amd64 GNU C Library: Shared libraries ii libcap-ng0:amd64 0.7.9-2.1build1 amd64 An alternate POSIX capabilities library ii libdebconfclient0:amd64 0.251ubuntu1 amd64 Debian Configuration Management System (C-implementation library) ii libevent-openssl-2.1-7:amd64 2.1.11-stable-1 amd64 Asynchronous event notification library (openssl) ii libkmod2:amd64 27-1ubuntu2.1 amd64 libkmod shared library ii libprocps8:amd64 2:3.3.16-1ubuntu2.3 amd64 library for accessing process information from /proc ii libssl1.1:amd64 1.1.1f-1ubuntu2.19 amd64 Secure Sockets Layer toolkit - shared libraries ii libunbound8:amd64 1.9.4-2ubuntu1.4 amd64 library implementing DNS resolution and validation ii libxmlsec1-openssl:amd64 1.2.28-2 amd64 Openssl engine for the XML security library ii lsb-base 11.1.0ubuntu2 all Linux Standard Base init script functionality ii lsb-release 11.1.0ubuntu2 all Linux Standard Base version reporting utility ii netbase 6.1 all Basic TCP/IP networking system ii octavia-common 6.2.2-0ubuntu1 all OpenStack Load Balancer as a Service - Common files ii openssl 1.1.1f-1ubuntu2.19 amd64 Secure Sockets Layer toolkit - cryptographic utility ii openvswitch-common 2.13.8-0ubuntu1.2 amd64 Open vSwitch common components ii openvswitch-switch 2.13.8-0ubuntu1.2 amd64 Open vSwitch switch implementations ii ovn-common 20.03.2-0ubuntu0.20.04.4 amd64 OVN common components ii procps 2:3.3.16-1ubuntu2.3 amd64 /proc file system utilities ii python3-alembic 1.1.0-1ubuntu1 all lightweight database migration tool for SQLAlchemy - Python module ii python3-babel 2.6.0+dfsg.1-1ubuntu2.2 all tools for internationalizing Python applications - Python 3.x ii python3-barbicanclient 4.10.0-0ubuntu1 all OpenStack Key Management API client - Python 3.x ii python3-castellan 3.0.1-0ubuntu1 all generic key manager interface for OpenStack - Python 3.x ii python3-cinderclient 1:7.0.0-0ubuntu1 all Python bindings to the OpenStack Volume API - Python 3.x ii python3-cotyledon 1.7.3-2 all framework for defining long-running services - Python 3.x ii python3-cryptography 2.8-3ubuntu0.1 amd64 Python library exposing cryptographic recipes and primitives (Python 3) ii python3-debconf 1.5.73 all interact with debconf from Python 3 ii python3-debtcollector 2.0.0-0ubuntu1 all collection of patterns to collect technical debt - Python 3.x ii python3-diskimage-builder 2.35.0-0ubuntu1 all image building tools for Openstack - Python 3.x ii python3-distro 1.4.0-1 all Linux OS platform information API ii python3-distro-info 0.23ubuntu1 all information about distributions' releases (Python 3 module) ii python3-flask 1.1.1-2ubuntu0.1 all micro web framework based on Werkzeug and Jinja2 - Python 3.x ii python3-futurist 2.1.0-0ubuntu1 all useful additions to futures, from the future - Python 3.x ii python3-glanceclient 1:3.1.1-0ubuntu1 all Client library for Openstack glance server - Python 3.x ii python3-gunicorn 20.0.4-3 all Event-based HTTP/WSGI server (Python 3 libraries) ii python3-jinja2 2.10.1-2 all small but fast and easy to use stand-alone template engine ii python3-jsonschema 3.2.0-0ubuntu2 all An(other) implementation of JSON Schema (Draft 3 and 4) - Python 3.x ii python3-keystoneauth1 4.0.0-0ubuntu1 all authentication library for OpenStack Identity - Python 3.x ii python3-keystonemiddleware 9.0.0-0ubuntu1 all Middleware for OpenStack Identity (Keystone) - Python 3.x ii python3-netaddr 0.7.19-3ubuntu1 all manipulation of various common network address notations (Python 3) ii python3-netifaces 0.10.4-1ubuntu4 amd64 portable network interface information - Python 3.x ii python3-neutron-lib 2.3.0-0ubuntu1 all Neutron shared routines and utilities - Python 3.x ii python3-neutronclient 1:7.1.1-0ubuntu1 all client API library for Neutron - Python 3.x ii python3-novaclient 2:17.0.0-0ubuntu1 all client library for OpenStack Compute API - 3.x ii python3-octavia 6.2.2-0ubuntu1 all OpenStack Load Balancer as a Service - Python libraries ii python3-octavia-lib 2.0.0-0ubuntu1 all Library to support Octavia provider drivers ii python3-openssl 19.0.0-1build1 all Python 3 wrapper around the OpenSSL library ii python3-openstackclient 5.2.0-0ubuntu1.20.04.1 all OpenStack Command-line Client - Python 3.x ii python3-openvswitch 2.13.8-0ubuntu1.2 all Python 3 bindings for Open vSwitch ii python3-oslo.config 1:8.0.2-0ubuntu1 all Common code for Openstack Projects (configuration API) - Python 3.x ii python3-oslo.context 1:3.0.2-0ubuntu1 all WSGI context helpers for OpenStack - Python 3.x ii python3-oslo.db 8.1.0-0ubuntu1 all database connectivity to the different backends and helper utils - Python 3.x ii python3-oslo.i18n 4.0.1-0ubuntu1 all Oslo Internationalization Utilities - Python 3.x ii python3-oslo.log 4.1.1-0ubuntu1 all OpenStack logging configuration library - Python 3.x ii python3-oslo.messaging 12.1.6-0ubuntu1 all oslo messaging library - Python 3.x ii python3-oslo.middleware 4.0.2-0ubuntu1 all various WSGI middleware components for OpenStack - Python 3.x ii python3-oslo.policy 3.1.0-0ubuntu1.1 all RBAC policy enforcement library for OpenStack - Python 3.x ii python3-oslo.reports 2.0.1-0ubuntu1 all reports serialized in various data types - Python 3.x ii python3-oslo.serialization 3.1.1-0ubuntu1 all utilities for serialization , especially JSON - Python 3.x ii python3-oslo.upgradecheck 1.0.1-0ubuntu1 all Common code for writing upgrade checks - Python 3 ii python3-oslo.utils 4.1.1-0ubuntu1.1 all set of utility functions for OpenStack - Python 3.x ii python3-ovsdbapp 1.1.0-0ubuntu2 all library for creating OVSDB applications - Python 3.x ii python3-pbr 5.4.5-0ubuntu1 all inject useful and sensible default behaviors into setuptools - Python 3.x ii python3-pecan 1.3.3-3ubuntu1 all WSGI object-dispatching web framework - Python 3.x ii python3-pyasn1 0.4.2-3build1 all ASN.1 library for Python (Python 3 module) ii python3-pyasn1-modules 0.2.1-0.2build1 all Collection of protocols modules written in ASN.1 language (Python 3) ii python3-pymysql 0.9.3-2ubuntu3 all Pure-Python MySQL Driver - Python 3.x ii python3-pyroute2 0.5.9-0ubuntu2 all Python3 Netlink library ii python3-requests 2.22.0-2ubuntu1 all elegant and simple HTTP library for Python3, built for human beings ii python3-requests-oauthlib 1.0.0-1.1build1 all module providing OAuthlib auth support for requests (Python 3) ii python3-requests-unixsocket 0.2.0-2 all Use requests to talk HTTP via a UNIX domain socket - Python 3.x ii python3-requestsexceptions 1.4.0-3 all import exceptions from bundled packages in requests. - Python 3.x ii python3-rfc3986 1.3.2-0ubuntu2 all validating URI references per RFC 3986 - Python 3.x ii python3-setproctitle:amd64 1.1.10-1ubuntu1 amd64 Setproctitle implementation for Python 3 ii python3-simplejson 3.16.0-2ubuntu2 amd64 simple, fast, extensible JSON encoder/decoder for Python 3.x ii python3-six 1.14.0-2 all Python 2 and 3 compatibility library (Python 3 interface) ii python3-sortedcontainers 2.1.0-2 all sorted container types: SortedList, SortedDict, and SortedSet (Python 3) ii python3-sqlalchemy 1.3.12+ds1-1ubuntu2 all SQL toolkit and Object Relational Mapper for Python 3 ii python3-sqlalchemy-ext 1.3.12+ds1-1ubuntu2 amd64 SQL toolkit and Object Relational Mapper for Python3 - C extension ii python3-sqlalchemy-utils 0.36.1-0ubuntu1 all various utility functions for SQLAlchemy - Python 3.x ii python3-stevedore 1:1.32.0-0ubuntu2 all manage dynamic plugins for Python applications - python3 ii python3-taskflow 4.1.0-0ubuntu1 all Taskflow structured state management library - Python 3.x ii python3-tenacity 6.0.0-0ubuntu2 all retry code until it succeeeds ii python3-webob 1:1.8.5-2 all Python module providing WSGI request and response objects (Python 3) ii python3-werkzeug 0.16.1+dfsg1-2ubuntu0.1 all collection of utilities for WSGI applications (Python 3.x) ii python3-wsme 0.9.3-3 all Web Services Made Easy: implement multi-protocol webservices - Python 3.x ii ssl-cert 1.0.39 all simple debconf wrapper for OpenSSL ii uuid-runtime 2.34-0.1ubuntu9.3 amd64 runtime components for the Universally Unique ID library