OS Ubuntu 16.04
Version 14.1.1 upgrading to stable/ocata
Docs - https://docs.openstack.org/developer/openstack-ansible/upgrade-guide/script-upgrade.html
Possibly duplicate of https://bugs.launchpad.net/openstack-ansible/+bug/1667130
On an environment with successful Tempest results of Stable/Newton, and upgrading to Stable/Ocata via the upgrade script, post upgrade tests fail. Nova logs report IncompatibleVersion errors, and all nova services are reporting running 14.1.1 versions. Other services correctly run the 15.0.0 version after upgrade.
http://paste.openstack.org/show/603195/
Restarting all nodes/containers then results in all services running the 15.0.0 version and Tempest tests running successfully.
The Ansible output of the upgrade does show nova containers were not restarted
TASK [Lxc container restart] ***************************************************
Friday 17 March 2017 15:09:03 -0500 (0:00:01.844) 0:07:59.862 **********
skipping: [infra01_nova_console_container-6c00742d]
skipping: [infra02_nova_console_container-34ac99c3]
skipping: [infra01_nova_conductor_container-bda26a60]
skipping: [infra03_nova_conductor_container-ad862a38]
skipping: [infra02_nova_conductor_container-11d4b383]
skipping: [infra01_nova_api_metadata_container-9a0788d7]
skipping: [infra03_nova_api_metadata_container-c8765b95]
skipping: [infra02_nova_api_metadata_container-7827b6fd]
skipping: [infra01_nova_api_os_compute_container-96f37e5e]
skipping: [infra03_nova_api_os_compute_container-2a1d8c4c]
skipping: [infra02_nova_api_os_compute_container-65538299]
skipping: [infra01_nova_cert_container-67530cee]
skipping: [infra03_nova_cert_container-c17f638e]
skipping: [infra02_nova_cert_container-6265c5e0]
skipping: [infra01_nova_scheduler_container-50325832]
skipping: [infra03_nova_scheduler_container-2225daf1]
skipping: [infra02_nova_scheduler_container-3446e7c8]
skipping: [compute02]
skipping: [compute03]
skipping: [compute01]
skipping: [compute06]
skipping: [compute07]
skipping: [compute04]
skipping: [compute05]
skipping: [compute08]
skipping: [compute09]
skipping: [infra01_nova_api_placement_container-b6acbeb3]
skipping: [infra03_nova_api_placement_container-c9e430a9]
skipping: [infra03_nova_console_container-3c16b6e4]
skipping: [infra02_nova_api_placement_container-f3bd6d65]
Full installation/upgrade logs can be found at (HUGE_FILE_ALERT):
http://172.99.106.115/jenkins/job/22Node_BME_Upgrade(danko)/131/console
the nova services within the containers should have restarted when the new set of venvs, config, and init scripts were dropped. If any of that didn't happen we'd be facing an issue with the nova service as found within the os_nova role. I could see on possibility where the handler maybe didn't fire? Was this an upgrade that ran into an issue mid-upgrade and was rerun?
As for the container restarts we don't restart all containers on all upgrade, in fact we do our best not to restart containers in an effort to maximize uptime. So the fact that the container restart was all skipped, in my mind, means that we did a decent job of not impacting uptime during the major migration.
As for the fact that all of the services were running the proper version once the containers were restarted, I'd have to suspect that there was something else that happened that caused the handler not to fire which could be a bug in the role or something else. Would it be possible to get the entire log for the run? Also have we confirmed this issue on multiple runs?