kolla-ansible

Comment 4 for bug 1833069

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-06-27: Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/665660
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=e6d2b92200d02715649d923b0ef2d6981905a6b9
Submitter: Zuul
Branch: master

commit e6d2b92200d02715649d923b0ef2d6981905a6b9
Author: Mark Goddard <email address hidden>
Date: Mon Jun 17 13:48:13 2019 +0100

Restart all nova services after upgrade

    During an upgrade, nova pins the version of RPC calls to the minimum
    seen across all services. This ensures that old services do not receive
    data they cannot handle. After the upgrade is complete, all nova
    services are supposed to be reloaded via SIGHUP to cause them to check
    again the RPC versions of services and use the new latest version which
    should now be supported by all running services.

    Due to a bug [1] in oslo.service, sending services SIGHUP is currently
    broken. We replaced the HUP with a restart for the nova_compute
    container for bug 1821362, but not other nova services. It seems we need
    to restart all nova services to allow the RPC version pin to be removed.

Testing in a Queens to Rocky upgrade, we find the following in the logs:

Automatically selected compute RPC version 5.0 from minimum service
version 30

However, the service version in Rocky is 35.

    There is a second issue in that it takes some time for the upgraded
    services to update the nova services database table with their new
    version. We need to wait until all nova-compute services have done this
    before the restart is performed, otherwise the RPC version cap will
    remain in place. There is currently no interface in nova available for
    checking these versions [2], so as a workaround we use a configurable
    delay with a default duration of 30 seconds. Testing showed it takes
    about 10 seconds for the version to be updated, so this gives us some
    headroom.

This change restarts all nova services after an upgrade, after a 30
second delay.

[1] https://bugs.launchpad.net/oslo.service/+bug/1715374
[2] https://bugs.launchpad.net/nova/+bug/1833542

    Change-Id: Ia6fc9011ee6f5461f40a1307b72709d769814a79
    Closes-Bug: #1833069
    Related-Bug: #1833542

Reviewed:  https://review.opendev.org/665660
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=e6d2b92200d02715649d923b0ef2d6981905a6b9
Submitter: Zuul
Branch:    master

commit e6d2b92200d02715649d923b0ef2d6981905a6b9
Author: Mark Goddard <mark@stackhpc.com>
Date:   Mon Jun 17 13:48:13 2019 +0100

Restart all nova services after upgrade
    
    During an upgrade, nova pins the version of RPC calls to the minimum
    seen across all services. This ensures that old services do not receive
    data they cannot handle. After the upgrade is complete, all nova
    services are supposed to be reloaded via SIGHUP to cause them to check
    again the RPC versions of services and use the new latest version which
    should now be supported by all running services.
    
    Due to a bug [1] in oslo.service, sending services SIGHUP is currently
    broken. We replaced the HUP with a restart for the nova_compute
    container for bug 1821362, but not other nova services. It seems we need
    to restart all nova services to allow the RPC version pin to be removed.
    
    Testing in a Queens to Rocky upgrade, we find the following in the logs:
    
    Automatically selected compute RPC version 5.0 from minimum service
    version 30
    
    However, the service version in Rocky is 35.
    
    There is a second issue in that it takes some time for the upgraded
    services to update the nova services database table with their new
    version. We need to wait until all nova-compute services have done this
    before the restart is performed, otherwise the RPC version cap will
    remain in place. There is currently no interface in nova available for
    checking these versions [2], so as a workaround we use a configurable
    delay with a default duration of 30 seconds. Testing showed it takes
    about 10 seconds for the version to be updated, so this gives us some
    headroom.
    
    This change restarts all nova services after an upgrade, after a 30
    second delay.
    
    [1] https://bugs.launchpad.net/oslo.service/+bug/1715374
    [2] https://bugs.launchpad.net/nova/+bug/1833542
    
    Change-Id: Ia6fc9011ee6f5461f40a1307b72709d769814a79
    Closes-Bug: #1833069
    Related-Bug: #1833542