nova-api fails to start during undercloud upgrade because httpd is already listening on port 8774

[Marius Cornea]

Undercloud upgrade from Newton to Ocata fails because nova-api service fails as httpd is already listening on port 8774:

2017-02-06 12:24:13,525 INFO: Error: /Stage[main]/Nova::Api/Nova::Generic_service[api]/Service[nova-api]: Failed to call refresh: Systemd restart for openstack-nova-api failed!
2017-02-06 12:24:13,525 INFO: journalctl log for openstack-nova-api:
2017-02-06 12:24:13,525 INFO: -- Logs begin at Tue 2017-01-31 14:13:51 UTC, end at Mon 2017-02-06 12:24:13 UTC. --
2017-02-06 12:24:13,526 INFO: Feb 06 12:24:08 undercloud-0.redhat.local systemd[1]: Stopping OpenStack Nova API Server...
2017-02-06 12:24:13,526 INFO: Feb 06 12:24:08 undercloud-0.redhat.local systemd[1]: Starting OpenStack Nova API Server...
2017-02-06 12:24:13,526 INFO: Feb 06 12:24:13 undercloud-0.redhat.local systemd[1]: openstack-nova-api.service: main process exited, code=exited, status=1/FAILURE
2017-02-06 12:24:13,526 INFO: Feb 06 12:24:13 undercloud-0.redhat.local systemd[1]: Failed to start OpenStack Nova API Server.
2017-02-06 12:24:13,526 INFO: Feb 06 12:24:13 undercloud-0.redhat.local systemd[1]: Unit openstack-nova-api.service entered failed state.
2017-02-06 12:24:13,526 INFO: Feb 06 12:24:13 undercloud-0.redhat.local systemd[1]: openstack-nova-api.service failed.
2017-02-06 12:24:13,526 INFO:
2017-02-06 12:24:13,527 INFO: Error: /Stage[main]/Nova::Api/Nova::Generic_service[api]/Service[nova-api]: Systemd restart for openstack-nova-api failed!
2017-02-06 12:24:13,527 INFO: journalctl log for openstack-nova-api:
2017-02-06 12:24:13,527 INFO: -- Logs begin at Tue 2017-01-31 14:13:51 UTC, end at Mon 2017-02-06 12:24:13 UTC. --
2017-02-06 12:24:13,527 INFO: Feb 06 12:24:08 undercloud-0.redhat.local systemd[1]: Stopping OpenStack Nova API Server...
2017-02-06 12:24:13,527 INFO: Feb 06 12:24:08 undercloud-0.redhat.local systemd[1]: Starting OpenStack Nova API Server...
2017-02-06 12:24:13,527 INFO: Feb 06 12:24:13 undercloud-0.redhat.local systemd[1]: openstack-nova-api.service: main process exited, code=exited, status=1/FAILURE
2017-02-06 12:24:13,527 INFO: Feb 06 12:24:13 undercloud-0.redhat.local systemd[1]: Failed to start OpenStack Nova API Server.
2017-02-06 12:24:13,527 INFO: Feb 06 12:24:13 undercloud-0.redhat.local systemd[1]: Unit openstack-nova-api.service entered failed state.
2017-02-06 12:24:13,528 INFO: Feb 06 12:24:13 undercloud-0.redhat.local systemd[1]: openstack-nova-api.service failed.
2017-02-06 12:24:13,528 INFO:

We can see in /var/log/nova/nova-api.log:

2017-02-06 12:24:35.105 25688 ERROR nova File "/usr/lib/python2.7/site-packages/eventlet/convenience.py", line 43, in listen
2017-02-06 12:24:35.105 25688 ERROR nova sock.bind(addr)
2017-02-06 12:24:35.105 25688 ERROR nova File "/usr/lib64/python2.7/socket.py", line 224, in meth
2017-02-06 12:24:35.105 25688 ERROR nova return getattr(self._sock,name)(*args)
2017-02-06 12:24:35.105 25688 ERROR nova error: [Errno 98] Address already in use

httpd is already binding on port 8774.

Later during the manifest run the 8774 port entry gets removed from /etc/httpd/conf/ports.conf:
2017-02-06 12:25:45,882 INFO: Notice: /Stage[main]/Apache/Concat[/etc/httpd/conf/ports.conf]/File[/etc/httpd/conf/ports.conf]/content: content changed '{md5}9fa93e8a204d9e6620060178f9202369' to '{md5}75666dfce1457d51f78b985fa0a03c98'
2017-02-06 12:25:45,889 INFO: Notice: /Stage[main]/Apache/File[/etc/httpd/conf/httpd.conf]/content: content changed '{md5}950fa8a069613e78f3be5af438321ac1' to '{md5}f1abcefe05f389de1abd19a13778e57a'

The undercloud upgrade fails in the end and the workaround to get it successfully complete is to rerun openstack undercloud upgrade.

[Brent Eagles (beagles)]

Does this happen if you stop the services first as described by http://docs.openstack.org/developer/tripleo-docs/installation/updating.html

[Marius Cornea (mcornea)]

I need to test it but I suspect it won't show up if httpd is stopped before. I'll add it to the upgrade docs as well if it works fine:
http://docs.openstack.org/developer/tripleo-docs/post_deployment/upgrade.html

[Marius Cornea (mcornea)]

I wasn't able to reproduce this when stopping httpd before running openstack undercloud upgrade.

[Emilien Macchi (emilienm)]

Fixed by https://github.com/openstack/tripleo-docs/commit/600ecec57997be2261183e028cace08260ec091d