openstack-upgrade action of cinder "conflicts" with nova-compute if deployed to same metal.

[Paul Goins]

Hi,

As noted on a separate bug (https://bugs.launchpad.net/charm-deployment-guide/+bug/1929091), when I recently tried doing an OpenStack upgrade following the charm deployment guide, I hit issues due to interactions between cinder-volume (cinder deployed onto the metals with enabled-services=volume) and nova-compute.

If I set action-managed-upgrades=true and updated openstack-origin on cinder-volume, then executed the upgrade-openstack action, the "apt-get update" and "apt-get upgrade" performed during this process ended up also upgrading the nova-compute packages. This meant that, when I went around to performing the upgrade on nova-compute, the actions claimed that things were already upgraded.

Likewise, if I did the same but upgraded nova-compute first, cinder-volume would claim that it has already been upgraded.

While I did compare the end result on my recent upgrade (stein->train) and saw no real difference between the packages installed with either order, this gave me pause as I was not sure if all actions related to the upgrade process were being done appropriately for whichever charm was upgraded last.

My concerns:

1. In a case of a "conflict" like this, are we sure that we're running all needed upgrade-related actions? It seems to me like we might not; do_action_managed_upgrade() in charmhelpers returns False in the "no upgrade available" case, and would avoid even calling the upgrade callback, which may result in e.g. skipping cinder DB migrations if nova-compute is upgraded first (if relevant for such cinder units).

2. Are there any issues or workarounds which should be added to the charm deployment guide to help address this?

Best Regards,
Paul Goins

[Andrea Ieri (aieri)]

This is probably caused by the do_openstack_upgrade function running a full apt upgrade after switching the sources:

hooks/nova_compute_utils.py
 857 │ def do_openstack_upgrade(configs=None):
 858 │ """Perform an uprade of cinder. Takes care of upgrading
 859 │ packages, rewriting configs + database migration and
 860 │ potentially any other post-upgrade actions.
 861 │
 862 │ :param configs: The charms main OSConfigRenderer object.
 863 │
 864 │ """
 865 │ new_src = config('openstack-origin')
 866 │ new_os_rel = get_os_codename_install_source(new_src)
 867 │
 868 │ juju_log('Performing OpenStack upgrade to %s.' % (new_os_rel))
 869 │
 870 │ configure_installation_source(new_src)
 871 │ dpkg_opts = [
 872 │ '--option', 'Dpkg::Options::=--force-confnew',
 873 │ '--option', 'Dpkg::Options::=--force-confdef',
 874 │ ]
 875 │ apt_update()
 876 │ apt_upgrade(options=dpkg_opts, fatal=True, dist=True)

Compare with ceph-osd, which only uses apt_install on the specific ceph packages during the upgrade:

lib/charms_ceph/utils.py
2680 │ def upgrade_osd(new_version, kick_function=None):
2681 │ """Upgrades the current OSD
2682 │
2683 │ :param new_version: str. The new version to upgrade to
2684 │ """
2685 │ if kick_function is None:
2686 │ kick_function = noop
2687 │
2688 │ current_version = get_version()
2689 │ status_set("maintenance", "Upgrading OSD")
2690 │ log("Current Ceph version is {}".format(current_version))
2691 │ log("Upgrading to: {}".format(new_version))
2692 │
2693 │ try:
2694 │ add_source(config('source'), config('key'))
2695 │ apt_update(fatal=True)
2696 │ except subprocess.CalledProcessError as err:
2697 │ log("Adding the Ceph sources failed with message: {}".format(
2698 │ err))
2699 │ status_set("blocked", "Upgrade to {} failed".format(new_version))
2700 │ sys.exit(1)
2701 │
2702 │ kick_function()
2703 │
2704 │ try:
2705 │ # Upgrade the packages before restarting the daemons.
2706 │ status_set('maintenance', 'Upgrading packages to %s' % new_version)
2707 │ apt_install(packages=determine_packages(), fatal=True)

The nova compute charm has the same problem.