There are two broken things testing upgrades locally, because the multinode upgrades job currently runs with selinux in permissive mode:
[shardy@localhost tripleo-heat-templates]$ grep -R setenforce ./*
./deployed-server/deployed-server-bootstrap-centos.sh:setenforce 0
After moving to httpd, cinder-api breaks like this:
[Thu Jan 19 14:01:25.845412 2017] [:error] [pid 155260] [remote 192.168.24.9:8] mod_wsgi (pid=155260): Exception occurred processing WSGI script '/var/www/cgi-bin/cinder/cinder-api'.
[Thu Jan 19 14:01:25.845428 2017] [:error] [pid 155260] [remote 192.168.24.9:8] Traceback (most recent call last):
[Thu Jan 19 14:01:25.845439 2017] [:error] [pid 155260] [remote 192.168.24.9:8] File "/var/www/cgi-bin/cinder/cinder-api", line 51, in <module>
[Thu Jan 19 14:01:25.845454 2017] [:error] [pid 155260] [remote 192.168.24.9:8] application = initialize_application()
[Thu Jan 19 14:01:25.845459 2017] [:error] [pid 155260] [remote 192.168.24.9:8] File "/usr/lib/python2.7/site-packages/cinder/wsgi/wsgi.py", line 42, in initialize_application
[Thu Jan 19 14:01:25.845466 2017] [:error] [pid 155260] [remote 192.168.24.9:8] logging.setup(CONF, "cinder")
[Thu Jan 19 14:01:25.845470 2017] [:error] [pid 155260] [remote 192.168.24.9:8] File "/usr/lib/python2.7/site-packages/oslo_log/log.py", line 269, in setup
[Thu Jan 19 14:01:25.845477 2017] [:error] [pid 155260] [remote 192.168.24.9:8] _setup_logging_from_conf(conf, product_name, version)
[Thu Jan 19 14:01:25.845480 2017] [:error] [pid 155260] [remote 192.168.24.9:8] File "/usr/lib/python2.7/site-packages/oslo_log/log.py", line 366, in _setup_logging_from_conf
[Thu Jan 19 14:01:25.845486 2017] [:error] [pid 155260] [remote 192.168.24.9:8] filelog = file_handler(logpath)
[Thu Jan 19 14:01:25.845492 2017] [:error] [pid 155260] [remote 192.168.24.9:8] File "/usr/lib64/python2.7/logging/handlers.py", line 392, in __init__
[Thu Jan 19 14:01:25.845724 2017] [:error] [pid 155260] [remote 192.168.24.9:8] logging.FileHandler.__init__(self, filename, mode, encoding, delay)
[Thu Jan 19 14:01:25.845751 2017] [:error] [pid 155260] [remote 192.168.24.9:8] File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__
[Thu Jan 19 14:01:25.846104 2017] [:error] [pid 155260] [remote 192.168.24.9:8] StreamHandler.__init__(self, self._open())
[Thu Jan 19 14:01:25.846126 2017] [:error] [pid 155260] [remote 192.168.24.9:8] File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open
[Thu Jan 19 14:01:25.846153 2017] [:error] [pid 155260] [remote 192.168.24.9:8] stream = open(self.baseFilename, self.mode)
[Thu Jan 19 14:01:25.846173 2017] [:error] [pid 155260] [remote 192.168.24.9:8] IOError: [Errno 13] Permission denied: '/var/log/cinder/cinder-api.log'
There's also some problem with glance-api, where it fails to validate a token, this is also fixed by turning off selinux.
This is a bug about N->O upgrades, which seems like it must have been fixed at this point. I'm going to close it, but feel free to reopen if I'm mistaken.