neutron_openvswitch_agent keeps restarting because of docker health checks

Bug #1982281 reported by Mohammed Daoudi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
New
Undecided
Unassigned

Bug Description

**Bug Report**

What happened:
I installed openstack using kolla-ansible using the multinode setup (2 machines). after a restart. all ovs related container failed. After removing the pid files they started working again except neutron_openvswitch_agent container which starts and gets stuck in the unhealthy state.

What you expected to happen:
neutron_openvswitch_agent is healthy

How to reproduce it (minimal and precise):
Restarted both openstack machines and the problem started occuring.

**Environment**:
* OS (e.g. from /etc/os-release): "Ubuntu 20.04 LTS"
* Kernel (e.g. `uname -a`):Linux controller 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
* Docker version if applicable (e.g. `docker version`): 20.10.17 (containerd 1.6.6 / runc 1.1.2 / docker-init 0.19.0)
* Kolla-Ansible version (e.g. `git head or tag or stable branch` or pip package version if using release): git+https://opendev.org/openstack/kolla-ansible@3d65a160d93558a33c36b74deb4e92ec4a6d94bb
* Docker image Install type (source/binary): binary
* Docker image distribution: quay.io/openstack.kolla/neutron-openvswitch-agent:master-ubuntu-focal
* Are you using official images from Docker Hub or self built? official
* If self built - Kolla version and environment used to build:
* Share your inventory file, globals.yml and other configuration files if relevant

```
globals.yml
---
workaround_ansible_issue_8743: yes
kolla_base_distro: "ubuntu"
kolla_internal_vip_address: "192.168.3.35"
network_interface: "eno1"
neutron_external_interface: "eno2"
nova_compute_virt_type: "kvm"
nova_console: "spice"
```

and

```
# docker logs are normal
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting /etc/neutron/neutron.conf
INFO:__main__:Copying /var/lib/kolla/config_files/neutron.conf to /etc/neutron/neutron.conf
INFO:__main__:Setting permission for /etc/neutron/neutron.conf
INFO:__main__:Deleting /etc/neutron/plugins/ml2/openvswitch_agent.ini
INFO:__main__:Copying /var/lib/kolla/config_files/openvswitch_agent.ini to /etc/neutron/plugins/ml2/openvswitch_agent.ini
INFO:__main__:Setting permission for /etc/neutron/plugins/ml2/openvswitch_agent.ini
INFO:__main__:Writing out command to execute
INFO:__main__:Setting permission for /var/log/kolla/neutron
INFO:__main__:Setting permission for /var/log/kolla/neutron/neutron-server.log
INFO:__main__:Setting permission for /var/log/kolla/neutron/neutron-l3-agent.log
INFO:__main__:Setting permission for /var/log/kolla/neutron/neutron-metadata-agent.log
INFO:__main__:Setting permission for /var/log/kolla/neutron/neutron-server.log.1
INFO:__main__:Setting permission for /var/log/kolla/neutron/neutron-openvswitch-agent.log.2.gz
INFO:__main__:Setting permission for /var/log/kolla/neutron/neutron-dhcp-agent.log
INFO:__main__:Setting permission for /var/log/kolla/neutron/neutron-openvswitch-agent.log
INFO:__main__:Setting permission for /var/log/kolla/neutron/neutron-openvswitch-agent.log.1
INFO:__main__:Setting permission for /var/log/kolla/neutron/dnsmasq.log
INFO:__main__:Setting permission for /var/log/kolla/neutron/neutron-openvswitch-agent.log.3.gz
INFO:__main__:Setting permission for /var/log/kolla/neutron/privsep-helper.log
++ cat /run_command
+ CMD='neutron-openvswitch-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini'
+ ARGS=
+ sudo kolla_copy_cacerts
+ [[ ! -n '' ]]
+ . kolla_extend_start
++ [[ ! -d /var/log/kolla/neutron ]]
+++ stat -c %a /var/log/kolla/neutron
++ [[ 2755 != \7\5\5 ]]
++ chmod 755 /var/log/kolla/neutron
++ [[ ubuntu == \c\e\n\t\o\s ]]
++ export UPDATE_ALTERNATIVES=/usr/bin/update-alternatives
++ UPDATE_ALTERNATIVES=/usr/bin/update-alternatives
++ /usr/bin/update-alternatives --display iptables
iptables - auto mode
  link best version is /usr/sbin/iptables-legacy
  link currently points to /usr/sbin/iptables-legacy
  link iptables is /usr/sbin/iptables
  slave iptables-restore is /usr/sbin/iptables-restore
  slave iptables-save is /usr/sbin/iptables-save
/usr/sbin/iptables-legacy - priority 20
  slave iptables-restore: /usr/sbin/iptables-legacy-restore
  slave iptables-save: /usr/sbin/iptables-legacy-save
/usr/sbin/iptables-nft - priority 10
  slave iptables-restore: /usr/sbin/iptables-nft-restore
  slave iptables-save: /usr/sbin/iptables-nft-save
++ KOLLA_LEGACY_IPTABLES=false
++ [[ false == \t\r\u\e ]]
++ sudo /usr/bin/update-alternatives --auto iptables
++ sudo /usr/bin/update-alternatives --auto ip6tables
++ . /usr/local/bin/kolla_neutron_extend_start
+ echo 'Running command: '\''neutron-openvswitch-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini'\'''
Running command: 'neutron-openvswitch-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini'
+ exec neutron-openvswitch-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini
+ sudo -E kolla_set_configs
```

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.