Queens: epmd process is not always started by systemd when initiating undercloud upgrade

[Damien Ciabrini]

In Queens, undercloud is non containerized and RabbitMQ is started via systemd.
systemd automatically starts epmd via another dedicated epmd.service.

[root@undercloud ~]# systemctl status epmd@*
● [e-mail] - Erlang Port Mapper Daemon
   Loaded: loaded (/usr/lib/systemd/system/epmd@.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-04-03 03:12:53 EDT; 2h 53min ago
 Main PID: 20510 (epmd)
   Status: "Processing port mapping requests..."
   CGroup: /system.slice/system-epmd.slice/[e-mail]
           └─20510 /usr/bin/epmd -systemd

While working on lp#1822120 [1], we found that we need to restart epmd and rabbitmq
on undercloud upgrade because of erlang 18 -> erlang 19 major upgrade that occurs
on P -> Q upgrade.

However the ownership of epmd is not always clear, as it is normally started by
systemd, but it can also be started by rabbitmq if no epmd is running at the time
rabbitmq needs it, for whatever reason [2,3]

So in order to perform a restart of epmd and rabbitmq, in [1] we bindly kill the
running epmd, which is not super selective and doesn't try to recover from cases
where epmd is started by rabbitmq.

[1] https://review.openstack.org/#/c/649194
[2] http://logs.openstack.org/87/649287/2/check/tripleo-ci-centos-7-undercloud-oooq/f427642/logs/undercloud/var/log/extra/services.txt.gz
[3] https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1653724

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/649901

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/649901
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=dc2f29ac9ba18f672964d25862fe7cd4907af36e
Submitter: Zuul
Branch: stable/queens

commit dc2f29ac9ba18f672964d25862fe7cd4907af36e
Author: Damien Ciabrini <email address hidden>
Date: Thu Apr 4 09:04:08 2019 +0200

    [Queens-only] Ensure epmd is spawned by systemd before redeploy

    On (re)deploy, we forcibly restart epmd and rabbit to ensure any
    upgrade of erlang or rabbitmq can be taken into account before
    puppet runs. But there is currently no guarantee as to which
    one of systemd or rabbitmq will restart epmd.

    This is an improvement over I6f486b0b70f19d8b4916ef500675c0739939e060,
    to selectively kill epmd and ensure the systemd service will always
    restart it, like it is supposed to.

    Change-Id: I82b419cfe4585a3f185687b8cc0a19bb534f7536
    Closes-Bug: #1823003

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/instack-undercloud 8.4.8 release.