Systemd restart for openstack-nova-api failed when upgrading undercloud from newton to ocata

[zenghui.shi]

when upgrading undercloud from newton to ocata, it failed to restart nova-api service, the error log looks similar with ​bug #1662167, but there is no httpd ERRORs in nova-api.log.

...

2017-12-21 09:22:02,244 INFO: ^[[1;31mError: /Stage[main]/Nova::Api/Nova::Generic_service[api]/Service[nova-api]: Failed to call refresh: Systemd restart for openstack-nova-api failed!
2017-12-21 09:22:02,244 INFO: journalctl log for openstack-nova-api:
2017-12-21 09:22:02,244 INFO: -- Logs begin at Thu 2017-12-21 07:10:40 UTC, end at Thu 2017-12-21 09:22:02 UTC. --
2017-12-21 09:22:02,244 INFO: Dec 21 09:21:56 undercloud-0.redhat.local systemd[1]: Stopping OpenStack Nova API Server...
2017-12-21 09:22:02,244 INFO: Dec 21 09:21:56 undercloud-0.redhat.local systemd[1]: Starting OpenStack Nova API Server...
2017-12-21 09:22:02,244 INFO: Dec 21 09:22:02 undercloud-0.redhat.local systemd[1]: openstack-nova-api.service: main process exited, code=exited, status=1/FAILURE
2017-12-21 09:22:02,245 INFO: Dec 21 09:22:02 undercloud-0.redhat.local systemd[1]: Failed to start OpenStack Nova API Server.
2017-12-21 09:22:02,245 INFO: Dec 21 09:22:02 undercloud-0.redhat.local systemd[1]: Unit openstack-nova-api.service entered failed state.
2017-12-21 09:22:02,245 INFO: Dec 21 09:22:02 undercloud-0.redhat.local systemd[1]: openstack-nova-api.service failed.
2017-12-21 09:22:02,245 INFO: ^[[0m
2017-12-21 09:22:02,245 INFO: ^[[1;31mError: /Stage[main]/Nova::Api/Nova::Generic_service[api]/Service[nova-api]: Systemd restart for openstack-nova-api failed!
2017-12-21 09:22:02,245 INFO: journalctl log for openstack-nova-api:
2017-12-21 09:22:02,245 INFO: -- Logs begin at Thu 2017-12-21 07:10:40 UTC, end at Thu 2017-12-21 09:22:02 UTC. --
2017-12-21 09:22:02,245 INFO: Dec 21 09:21:56 undercloud-0.redhat.local systemd[1]: Stopping OpenStack Nova API Server...
2017-12-21 09:22:02,245 INFO: Dec 21 09:21:56 undercloud-0.redhat.local systemd[1]: Starting OpenStack Nova API Server...
2017-12-21 09:22:02,246 INFO: Dec 21 09:22:02 undercloud-0.redhat.local systemd[1]: openstack-nova-api.service: main process exited, code=exited, status=1/FAILURE
2017-12-21 09:22:02,246 INFO: Dec 21 09:22:02 undercloud-0.redhat.local systemd[1]: Failed to start OpenStack Nova API Server.
2017-12-21 09:22:02,246 INFO: Dec 21 09:22:02 undercloud-0.redhat.local systemd[1]: Unit openstack-nova-api.service entered failed state.
2017-12-21 09:22:02,246 INFO: Dec 21 09:22:02 undercloud-0.redhat.local systemd[1]: openstack-nova-api.service failed.

...

2017-12-21 09:29:45,323 INFO: Notice: Applied catalog in 653.39 seconds
2017-12-21 09:29:56,938 INFO: + rc=6
2017-12-21 09:29:56,938 INFO: + set -e
2017-12-21 09:29:56,939 INFO: + echo 'puppet apply exited with exit code 6'
2017-12-21 09:29:56,939 INFO: puppet apply exited with exit code 6
2017-12-21 09:29:56,939 INFO: + '[' 6 '!=' 2 -a 6 '!=' 0 ']'
2017-12-21 09:29:56,939 INFO: + exit 6
2017-12-21 09:29:56,939 INFO: [2017-12-21 09:29:56,939] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]
2017-12-21 09:29:56,939 INFO:
2017-12-21 09:29:56,939 INFO: [2017-12-21 09:29:56,939] (os-refresh-config) [ERROR] Aborting...
2017-12-21 09:29:56,949 ERROR:
#############################################################################
Undercloud upgrade failed.

Reason: os-refresh-config failed. See log for details.

See the previous output for details about what went wrong. The full install
log can be found at /home/stack/.instack/install-undercloud.log.

#############################################################################

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1485, in install
    _run_orc(instack_env)
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1208, in _run_orc
    _run_live_command(args, instack_env, 'os-refresh-config')
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 558, in _run_live_command
    raise RuntimeError('%s failed. See log for details.' % name)
RuntimeError: os-refresh-config failed. See log for details.
Command 'instack-upgrade-undercloud' returned non-zero exit status 1

[root@undercloud-0 nova]# rpm -qa | grep nova
openstack-nova-conductor-15.0.9-0.20171219232100.4f0ff43.el7.centos.noarch
python2-novaclient-7.1.2-0.20170621093845.2e2317b.el7.centos.noarch
puppet-nova-10.4.2-0.20171127233709.eb1fafa.el7.centos.noarch
openstack-nova-common-15.0.9-0.20171219232100.4f0ff43.el7.centos.noarch
openstack-nova-api-15.0.9-0.20171219232100.4f0ff43.el7.centos.noarch
python-nova-15.0.9-0.20171219232100.4f0ff43.el7.centos.noarch
openstack-nova-scheduler-15.0.9-0.20171219232100.4f0ff43.el7.centos.noarch
openstack-nova-compute-15.0.9-0.20171219232100.4f0ff43.el7.centos.noarch
openstack-nova-placement-api-15.0.9-0.20171219232100.4f0ff43.el7.centos.noarch
openstack-nova-cert-15.0.9-0.20171219232100.4f0ff43.el7.centos.noarch

[root@undercloud-0 nova]# yum repolist
Loaded plugins: fastestmirror, priorities
Loading mirror speeds from cached hostfile
 * base: reflector.westga.edu
 * extras: mirrors.sorengard.com
 * updates: mirrors.advancedhosters.com
1426 packages excluded due to repository priority protections
repo id repo name status
base/7/x86_64 CentOS-7 - Base 9,591
centos-ceph-jewel/7/x86_64 CentOS-7 - Ceph Jewel 88
delorean-ocata delorean-python-django-openstack-auth-e111d8cfcb0865298ccc4150bccb2f993cc0350 606+267
delorean-ocata-testing/x86_64 dlrn-ocata-testing 1,048+1,426
extras/7/x86_64 CentOS-7 - Extras 327
rdo-qemu-ev/x86_64 RDO CentOS-7 - QEMU EV 39
updates/7/x86_64 CentOS-7 - Updates 1,573

[Alex Schultz (alex-schultz)]

Ocata has been EOL for a while