Queens: epmd process is not always started by systemd when initiating undercloud upgrade

Bug #1823003 reported by Damien Ciabrini
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Triaged
Medium
Damien Ciabrini

Bug Description

In Queens, undercloud is non containerized and RabbitMQ is started via systemd.
systemd automatically starts epmd via another dedicated epmd.service.

[root@undercloud ~]# systemctl status epmd@*
● epmd@0.0.0.0.service - Erlang Port Mapper Daemon
   Loaded: loaded (/usr/lib/systemd/system/epmd@.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-04-03 03:12:53 EDT; 2h 53min ago
 Main PID: 20510 (epmd)
   Status: "Processing port mapping requests..."
   CGroup: /system.slice/system-epmd.slice/epmd@0.0.0.0.service
           └─20510 /usr/bin/epmd -systemd

While working on lp#1822120 [1], we found that we need to restart epmd and rabbitmq
on undercloud upgrade because of erlang 18 -> erlang 19 major upgrade that occurs
on P -> Q upgrade.

However the ownership of epmd is not always clear, as it is normally started by
systemd, but it can also be started by rabbitmq if no epmd is running at the time
rabbitmq needs it, for whatever reason [2,3]

So in order to perform a restart of epmd and rabbitmq, in [1] we bindly kill the
running epmd, which is not super selective and doesn't try to recover from cases
where epmd is started by rabbitmq.

[1] https://review.openstack.org/#/c/649194
[2] http://logs.openstack.org/87/649287/2/check/tripleo-ci-centos-7-undercloud-oooq/f427642/logs/undercloud/var/log/extra/services.txt.gz
[3] https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1653724

Changed in tripleo:
milestone: none → stein-rc1
status: New → Triaged
tags: added: queens-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to instack-undercloud (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/649901

Changed in tripleo:
milestone: stein-rc1 → train-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to instack-undercloud (stable/queens)

Reviewed: https://review.openstack.org/649901
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=dc2f29ac9ba18f672964d25862fe7cd4907af36e
Submitter: Zuul
Branch: stable/queens

commit dc2f29ac9ba18f672964d25862fe7cd4907af36e
Author: Damien Ciabrini <email address hidden>
Date: Thu Apr 4 09:04:08 2019 +0200

    [Queens-only] Ensure epmd is spawned by systemd before redeploy

    On (re)deploy, we forcibly restart epmd and rabbit to ensure any
    upgrade of erlang or rabbitmq can be taken into account before
    puppet runs. But there is currently no guarantee as to which
    one of systemd or rabbitmq will restart epmd.

    This is an improvement over I6f486b0b70f19d8b4916ef500675c0739939e060,
    to selectively kill epmd and ensure the systemd service will always
    restart it, like it is supposed to.

    Change-Id: I82b419cfe4585a3f185687b8cc0a19bb534f7536
    Closes-Bug: #1823003

tags: added: in-stable-queens
Changed in tripleo:
milestone: train-1 → train-2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/instack-undercloud 8.4.8

This issue was fixed in the openstack/instack-undercloud 8.4.8 release.

Changed in tripleo:
milestone: train-2 → train-3
Changed in tripleo:
milestone: train-3 → ussuri-1
Changed in tripleo:
milestone: ussuri-1 → ussuri-2
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-2 → ussuri-3
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-3 → ussuri-rc3
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-rc3 → victoria-1
Changed in tripleo:
milestone: victoria-1 → victoria-3
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.