nova-api fails to start during undercloud upgrade because httpd is already listening on port 8774

Bug #1662167 reported by Marius Cornea on 2017-02-06
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Emilien Macchi

Bug Description

Undercloud upgrade from Newton to Ocata fails because nova-api service fails as httpd is already listening on port 8774:

2017-02-06 12:24:13,525 INFO: Error: /Stage[main]/Nova::Api/Nova::Generic_service[api]/Service[nova-api]: Failed to call refresh: Systemd restart for openstack-nova-api failed!
2017-02-06 12:24:13,525 INFO: journalctl log for openstack-nova-api:
2017-02-06 12:24:13,525 INFO: -- Logs begin at Tue 2017-01-31 14:13:51 UTC, end at Mon 2017-02-06 12:24:13 UTC. --
2017-02-06 12:24:13,526 INFO: Feb 06 12:24:08 undercloud-0.redhat.local systemd[1]: Stopping OpenStack Nova API Server...
2017-02-06 12:24:13,526 INFO: Feb 06 12:24:08 undercloud-0.redhat.local systemd[1]: Starting OpenStack Nova API Server...
2017-02-06 12:24:13,526 INFO: Feb 06 12:24:13 undercloud-0.redhat.local systemd[1]: openstack-nova-api.service: main process exited, code=exited, status=1/FAILURE
2017-02-06 12:24:13,526 INFO: Feb 06 12:24:13 undercloud-0.redhat.local systemd[1]: Failed to start OpenStack Nova API Server.
2017-02-06 12:24:13,526 INFO: Feb 06 12:24:13 undercloud-0.redhat.local systemd[1]: Unit openstack-nova-api.service entered failed state.
2017-02-06 12:24:13,526 INFO: Feb 06 12:24:13 undercloud-0.redhat.local systemd[1]: openstack-nova-api.service failed.
2017-02-06 12:24:13,526 INFO:
2017-02-06 12:24:13,527 INFO: Error: /Stage[main]/Nova::Api/Nova::Generic_service[api]/Service[nova-api]: Systemd restart for openstack-nova-api failed!
2017-02-06 12:24:13,527 INFO: journalctl log for openstack-nova-api:
2017-02-06 12:24:13,527 INFO: -- Logs begin at Tue 2017-01-31 14:13:51 UTC, end at Mon 2017-02-06 12:24:13 UTC. --
2017-02-06 12:24:13,527 INFO: Feb 06 12:24:08 undercloud-0.redhat.local systemd[1]: Stopping OpenStack Nova API Server...
2017-02-06 12:24:13,527 INFO: Feb 06 12:24:08 undercloud-0.redhat.local systemd[1]: Starting OpenStack Nova API Server...
2017-02-06 12:24:13,527 INFO: Feb 06 12:24:13 undercloud-0.redhat.local systemd[1]: openstack-nova-api.service: main process exited, code=exited, status=1/FAILURE
2017-02-06 12:24:13,527 INFO: Feb 06 12:24:13 undercloud-0.redhat.local systemd[1]: Failed to start OpenStack Nova API Server.
2017-02-06 12:24:13,527 INFO: Feb 06 12:24:13 undercloud-0.redhat.local systemd[1]: Unit openstack-nova-api.service entered failed state.
2017-02-06 12:24:13,528 INFO: Feb 06 12:24:13 undercloud-0.redhat.local systemd[1]: openstack-nova-api.service failed.
2017-02-06 12:24:13,528 INFO:

We can see in /var/log/nova/nova-api.log:

2017-02-06 12:24:35.105 25688 ERROR nova File "/usr/lib/python2.7/site-packages/eventlet/convenience.py", line 43, in listen
2017-02-06 12:24:35.105 25688 ERROR nova sock.bind(addr)
2017-02-06 12:24:35.105 25688 ERROR nova File "/usr/lib64/python2.7/socket.py", line 224, in meth
2017-02-06 12:24:35.105 25688 ERROR nova return getattr(self._sock,name)(*args)
2017-02-06 12:24:35.105 25688 ERROR nova error: [Errno 98] Address already in use

httpd is already binding on port 8774.

Later during the manifest run the 8774 port entry gets removed from /etc/httpd/conf/ports.conf:
2017-02-06 12:25:45,882 INFO: Notice: /Stage[main]/Apache/Concat[/etc/httpd/conf/ports.conf]/File[/etc/httpd/conf/ports.conf]/content: content changed '{md5}9fa93e8a204d9e6620060178f9202369' to '{md5}75666dfce1457d51f78b985fa0a03c98'
2017-02-06 12:25:45,889 INFO: Notice: /Stage[main]/Apache/File[/etc/httpd/conf/httpd.conf]/content: content changed '{md5}950fa8a069613e78f3be5af438321ac1' to '{md5}f1abcefe05f389de1abd19a13778e57a'

The undercloud upgrade fails in the end and the workaround to get it successfully complete is to rerun openstack undercloud upgrade.

Marius Cornea (mcornea) on 2017-02-06
Changed in tripleo:
status: New → Confirmed
Marius Cornea (mcornea) on 2017-02-06
Changed in tripleo:
status: Confirmed → Triaged
importance: Undecided → Critical
milestone: none → ocata-rc1
Brent Eagles (beagles) wrote :

Does this happen if you stop the services first as described by http://docs.openstack.org/developer/tripleo-docs/installation/updating.html

Marius Cornea (mcornea) wrote :

I need to test it but I suspect it won't show up if httpd is stopped before. I'll add it to the upgrade docs as well if it works fine:
http://docs.openstack.org/developer/tripleo-docs/post_deployment/upgrade.html

Marius Cornea (mcornea) wrote :

I wasn't able to reproduce this when stopping httpd before running openstack undercloud upgrade.

Emilien Macchi (emilienm) wrote :
Changed in tripleo:
status: Triaged → Fix Released
assignee: nobody → Emilien Macchi (emilienm)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers