Nova/Neutron Playbook run on baremetal disables keystone service

[Bjoern]

On Stein I ran nova/neutron playbooks to rollout changes onto a baremetal cloud, I noticed that
apache2 (on Ubuntu) was no longer installed and/or enabled (depending with play you run) which takes down Keystone (nginx), Horizon (apache2) and aodh as some of these services not using wsgi via nginx. At this point we would have to disable apache2 uninstallation/stop in a baremetal case or move these to nginx too. But I don't know why these services have not been moved to nginx yet.

Additionally I noticed that nginx-full was updated/installed during os-nova-install run which maybe another reason apache turned off.

[Bjoern (bjoern-t)]

Today I ran a upgrade from 19.0 stein to 19.0.4 and can confirm the issues where roles collide with uwsgi/apache2 management on a bare metal deployment.

I was running up to os-keystone install and noticed that keystone and stopped working in the process along with apache2 stopped which kills horizon:

At this point it seems baremetal is not a suitable deployment options due to these service availability issues.

Packages changes up to os-keystone install:

Start-Date: 2019-10-21 16:05:59
Commandline: /usr/bin/apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold install apt-transport-https dmeventd lvm2 software-properties-common
Upgrade: python3-software-properties:amd64 (0.96.24.32.9, 0.96.24.32.11), dmeventd:amd64 (2:1.02.145-4.1ubuntu3, 2:1.02.145-4.1ubuntu3.18.04.1), lvm2:amd64 (2.02.176-4.1ubuntu3, 2.02.176-4.1ubuntu3.18.04.1), apt-transport-https:amd64 (1.6.11, 1.6.12), software-properties-common:amd64 (0.96.24.32.9, 0.96.24.32.11)
End-Date: 2019-10-21 16:06:35

Start-Date: 2019-10-21 16:24:04
Commandline: /usr/bin/apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold install nginx-extras
Install: libnginx-mod-http-fancyindex:amd64 (1.14.0-0ubuntu1.6, automatic), libhiredis0.13:amd64 (0.13.3-2.2, automatic), libnginx-mod-http-headers-more-filter:amd64 (1.14.0-0ubuntu1.6, automatic), libnginx-mod-http-perl:amd64 (1.14.0-0ubuntu1.6, automatic), libnginx-mod-http-uploadprogress:amd64 (1.14.0-0ubuntu1.6, automatic), libnginx-mod-nchan:amd64 (1.14.0-0ubuntu1.6, automatic), libnginx-mod-http-cache-purge:amd64 (1.14.0-0ubuntu1.6, automatic), libluajit-5.1-2:amd64 (2.1.0~beta3+dfsg-5.1, automatic), nginx-extras:amd64 (1.14.0-0ubuntu1.6), libluajit-5.1-common:amd64 (2.1.0~beta3+dfsg-5.1, automatic), libnginx-mod-http-lua:amd64 (1.14.0-0ubuntu1.6, automatic), libnginx-mod-http-ndk:amd64 (1.14.0-0ubuntu1.6, automatic)
Remove: nginx-full:amd64 (1.14.0-0ubuntu1.6)
End-Date: 2019-10-21 16:24:09

Start-Date: 2019-10-21 16:39:43
Commandline: /usr/bin/apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold install erlang-base-hipe erlang-nox
Upgrade: erlang-public-key:amd64 (1:20.3.8.21-1, 1:20.3.8.23-1), erlang-corba:amd64 (1:20.3.8.21-1, 1:20.3.8.23-1), erlang-os-mon:amd64 (1:20.3.8.21-1, 1:20.3.8.23-1), erlang-erl-docgen:amd64 (1:20.3.8.21-1, 1:20.3.8.23-1), erlang-syntax-tools:amd64 (1:20.3.8.21-1, 1:20.3.8.23-1), erlang-tools:amd64 (1:20.3.8.21-1, 1:20.3.8.23-1), erlang-eunit:amd64 (1:20.3.8.21-1, 1:20.3.8.23-1), erlang-diameter:amd64 (1:20.3.8.21-1, 1:20.3.8.23-1), erlang-edoc:amd64 (1:20.3.8.21-1, 1:20.3.8.23-1), erlang-runtime-tools:amd64 (1:20.3.8.21-1, 1:20.3.8.23-1), erlang-nox:amd64 (1:20.3.8.21-1, 1:20.3.8.23-1), erlang-odbc:amd64 (1:20.3.8.21-1, 1:20.3.8.23-1), erlang-xmerl:amd64 (1:20.3.8.21-1, 1:20.3.8.23-1), erlang-snmp:amd64 (1:20.3.8.21-1, 1:20.3.8.23-1), erlang-ssh:amd64 (1:20.3.8.21-1, 1:20.3.8.23-1), erlang-ssl:amd64 (1:20.3.8.21-1, 1:20.3.8.23-1), erlang-percept:amd64 (1:20.3.8.21-1, 1:20.3.8.23-1), erlang-eldap:amd64 (1:20.3.8.21-1, 1:20.3.8.23-1), erlang-crypto:amd64 (1:20.3.8.21-1, 1:20.3.8.23-1), erlang-mnesia:amd64 (1:20.3.8.21-1, 1:20.3.8....

[Bjoern (bjoern-t)]

Keystone likely went down due to the nginx-full during the install

[Jonathan Rosser (jrosser)]

The immediate issue you see is different roles using different nginx packages

keystone nginx-full https://opendev.org/openstack/openstack-ansible-os_keystone/src/branch/master/vars/debian.yml#L53

repo_server nginx-extras https://opendev.org/openstack/openstack-ansible-repo_server/src/branch/master/vars/debian.yml#L26

The first trivial patch would be to standardise on nginx-extras, the different packages being explained here https://askubuntu.com/questions/553937/what-is-the-difference-between-the-core-full-extras-and-light-packages-for-ngi

I can't see any reference to nginx in the os_nova code, can you point to where that is?

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/693033

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693033
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=461245350b52d9ba6edc0ac474c6c6d2cb16ea3e
Submitter: Zuul
Branch: master

commit 461245350b52d9ba6edc0ac474c6c6d2cb16ea3e
Author: Dave Wilde (d34dh0r453) <email address hidden>
Date: Tue Nov 5 09:42:22 2019 -0600

    Standardize on nginx-extras

    The use of nginx-full causes a service restart on package update
    which brings down the keystone endpoints.

    Change-Id: Ic9cc341edb6f2f0ba76bd301c9782fbcc5951544
    Related-Bug: 1847395

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/693903

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693903
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=bcca76a29eedcd7608c2d1e0e460b9850bdb00cf
Submitter: Zuul
Branch: stable/train

commit bcca76a29eedcd7608c2d1e0e460b9850bdb00cf
Author: Dave Wilde (d34dh0r453) <email address hidden>
Date: Tue Nov 5 09:42:22 2019 -0600

    Standardize on nginx-extras

    The use of nginx-full causes a service restart on package update
    which brings down the keystone endpoints.

    Depends-On: https://review.opendev.org/694759/
    Change-Id: Ic9cc341edb6f2f0ba76bd301c9782fbcc5951544
    Related-Bug: 1847395
    (cherry picked from commit 461245350b52d9ba6edc0ac474c6c6d2cb16ea3e)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/695974

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/695974
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=65292ca2d0a24721546b9b2e577099f0b85c7e43
Submitter: Zuul
Branch: stable/stein

commit 65292ca2d0a24721546b9b2e577099f0b85c7e43
Author: Dave Wilde (d34dh0r453) <email address hidden>
Date: Tue Nov 5 09:42:22 2019 -0600

    Standardize on nginx-extras

    The use of nginx-full causes a service restart on package update
    which brings down the keystone endpoints.

    Change-Id: Ic9cc341edb6f2f0ba76bd301c9782fbcc5951544
    Related-Bug: 1847395
    (cherry picked from commit 461245350b52d9ba6edc0ac474c6c6d2cb16ea3e)