Neutron Open vSwitch Agent hangs because of Sudo

[Mathias Ewald]

We deployed our environment with Kolla and noticed that services hang when used or don't even appear on RabbitMQ. Looking into the the container we would situations like this one:

(neutron-openvswitch-agent)[root@net01 /]# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
neutron 1 0.0 0.3 129204 56856 ? Ss+ 13:31 0:01 /usr/bin/python /usr/bin/neutron-openvswitch-agent --config-file /etc/neutron/neu
root 20 0.0 0.0 44988 3060 ? S+ 13:31 0:00 sudo neutron-rootwrap /etc/neutron/rootwrap.conf ovs-vsctl --timeout=10 --oneline
neutron 21 0.0 0.0 18160 3284 ? Ss+ 13:56 0:00 /bin/bash
root 33 0.0 0.0 18176 3312 ? Ss 13:56 0:00 /bin/bash
root 59 0.0 0.0 15568 2224 ? R+ 13:56 0:00 ps aux
(neutron-openvswitch-agent)[root@net01 /]#

The believe neutron-openvswitch-agent runs "sudo neutron-rootwrap ..." which never returns as it hangs there asking for password. Looking at /etc/sudoers.d/neutron_sudoers, we found this:

neutron ALL = (root) NOPASSWD: /var/lib/kolla/venv/bin/neutron-rootwrap /etc/neutron/rootwrap.conf *

Looks like "sudo neutron-rootwrap" as executed by the service does not match that line. Changing the line to

neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf *

solves the problem.

[Mathias Ewald (mewald)]

Ok, so here is some more information:

1. The file /etc/sudoers.d/neutron_sudoers shouldn't even exist in my opinion. I did install_type of binary, and the contents of this file look like they only apply for source type installation.

2. I thought I was doing something wrong during building of my images that lead to those problems, so I decided to use images on docker hub. I pulled kolla/centos-binary-neutron-openvswitch-agent and took a look at the sudoers configuration without deploying it do any node:

root@infra00:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
kolla/centos-binary-neutron-openvswitch-agent 2.0.1 4a9b5164b8f5 13 days ago 824.5 MB
root@infra00:~# docker run -it -u root 4a9b5164b8f5 /bin/bash
()[root@4c046c260363 /]# cd /etc/sudoers.d/
()[root@4c046c260363 sudoers.d]# ls -l
total 8
-r--r----- 1 root root 207 May 12 11:29 neutron
-r--r----- 1 root root 101 Jan 1 1970 neutron_sudoers
()[root@4c046c260363 sudoers.d]# cat neutron
Defaults:neutron !requiretty

neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf *
neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
()[root@4c046c260363 sudoers.d]# cat neutron_sudoers
neutron ALL = (root) NOPASSWD: /var/lib/kolla/venv/bin/neutron-rootwrap /etc/neutron/rootwrap.conf *
()[root@4c046c260363 sudoers.d]#

The image was configured with two sudoers files that relate to neutron: /etc/sudoers.d/neutron_sudoers with the same content as shown above and /etc/sudoers.d/neutron with the configuration that would allow the neutron user to execute "/usr/bin/neutron-rootwrap" which is exactly what is happening.

I also checked the ubuntu image:

root@infra00:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
kolla/ubuntu-binary-neutron-openvswitch-agent 2.0.1 fde0ee25afe8 13 days ago 418.5 MB
kolla/centos-binary-neutron-openvswitch-agent 2.0.1 4a9b5164b8f5 13 days ago 824.5 MB
root@infra00:~# docker run -it -u root fde0ee25afe8 /bin/bash
()[root@d6190afa6ad6 /]# ls -l /etc/sudoers.d/
total 8
-r--r----- 1 root root 958 Aug 27 2015 README
-r--r----- 1 root root 101 Jan 1 1970 neutron_sudoers
()[root@d6190afa6ad6 /]# cat /etc/sudoers.d/neutron_sudoers
neutron ALL = (root) NOPASSWD: /var/lib/kolla/venv/bin/neutron-rootwrap /etc/neutron/rootwrap.conf *
()[root@d6190afa6ad6 /]#

The ubuntu image does not have /etc/sudoers/neutron. I deployed my environment with ubuntu images again, and manuelly added that file to all neutron containers that use neutron-rootwrap. Since then everything works!

[Michał Jastrzębski (inc007)]

You are using mitaka or master? In master neutron sudoers.d looks differetly, it has kolla_neutron_sudoers. However contents looks like one you described.

Please do $which neutron-rootwrap -> this should point to one from sudoers.

PS fresh install of ubuntu on ubuntu 16 works like charm

[Mathias Ewald (mewald)]

Hi Michal, I was using stable/mitaka branch. The file you are referring to contains

neutron ALL = (root) NOPASSWD: /var/lib/kolla/venv/bin/neutron-rootwrap /etc/neutron/rootwrap.conf *

right? And it is kolla/docker/neutron/neutron-base/neutron_sudoers in the git repo!? As described above, I found out that my installation uses "/usr/bin/neutron-rootwrap" to that sudoers file doesn't match. I believe that /var/lib/kolla/venv/... is only used if Kolla builds containers with install_type = source. Did you run the test with install_type = source?

Here is my kolla-build.conf:

[DEFAULT]
base = ubuntu
base_tag = 14.04
namespace = meshstack
profile = default
push = true
registry = 10.242.1.10:4000
install_type = binary
tag = 2.0.2.0
maintainer = evoila GmbH

And my global.yml:

---
config_strategy: "COPY_ALWAYS"
kolla_base_distro: "ubuntu"
kolla_install_type: "binary"
openstack_release: "2.0.1"
kolla_internal_vip_address: "10.242.7.100"
kolla_internal_fqdn: "{{ kolla_internal_vip_address }}"
kolla_external_vip_address: "{{ kolla_internal_vip_address }}"
kolla_external_fqdn: "{{ kolla_external_vip_address }}"
neutron_plugin_agent: "openvswitch"
openstack_logging_debug: "True"
nova_console: "novnc"
enable_central_logging: "yes"
enable_ceph: "no"
enable_ceph_rgw: "no"
enable_cinder: "no"
enable_heat: "yes"
enable_horizon: "yes"
enable_ironic: "no"
enable_magnum: "no"
enable_manila: "no"
enable_mistral: "no"
enable_mongodb: "no"
enable_murano: "no"
enable_swift: "no"

[Dennis Mueller (dennismueller)]

Hello there, I have the same issue. When I made a all-in-one kolla-ansible deploy with self-build docker containers, the neutron_* containers hangs at neutron_rootwrap. In the containers there was no neutron-rootwrap /var/lib/kolla/venv/bin, but in /usr/bin. So the solution from Mathias helps.

I followed the Quickstart Guide (http://docs.openstack.org/developer/kolla/quickstart.html).
My system is an Ubuntu 14.04.4. I done a kolla build with this kolla-build.conf:
[DEFAULT]
base = ubuntu
base_tag = 14.04
base_image = ubuntu
tag = 2.0.1

After that my kolla-ansible deploy was done with this global.yaml:
---
kolla_base_distro: "ubuntu"
kolla_internal_vip_address: "172.16.241.254"
network_interface: "eth0"
enable_central_logging: "no"
enable_ceph: "no"
enable_ceph_rgw: "no"
enable_cinder: "no"
enable_heat: "yes"
enable_horizon: "yes"
enable_ironic: "no"
enable_magnum: "no"
enable_manila: "no"
enable_mistral: "no"
enable_mongodb: "yes"
enable_murano: "no"
enable_swift: "no"

I used ansible==1.9.6 because of the kolla-ansible prechecks, and have no docker registry configured.

[Dennis Mueller (dennismueller)]

Attachment:

I checked out this code https://github.com/openstack/kolla.git and switched to stable/mitaka branch.

The global.yaml looks like this:
---
kolla_base_distro: "ubuntu"
openstack_release: "2.0.1"
kolla_internal_vip_address: "172.16.241.254"
network_interface: "eth0"
enable_central_logging: "no"
enable_ceph: "no"
enable_ceph_rgw: "no"
enable_cinder: "no"
enable_heat: "yes"
enable_horizon: "yes"
enable_ironic: "no"
enable_magnum: "no"
enable_manila: "no"
enable_mistral: "no"
enable_mongodb: "yes"
enable_murano: "no"
enable_swift: "no"

[Christian Berendt (berendt)]

Changing priority to medium.

inc0, any news here?