Train - Undercloud minor update fail at Start containers for step 3 using paunch

[Harald Jensås]

RuntimeError: Failing to apply action exec for container: keystone_bootstrap"

python2-tripleo-common-11.3.3-0.20200518054928.4394dac.el7.noarch
python2-tripleoclient-12.3.2-0.20200515171419.fbfd309.el7.noarch
openstack-tripleo-common-11.3.3-0.20200518054928.4394dac.el7.noarch

[root@undercloud stdouts]# tail /var/log/containers/stdouts/keystone.log
2020-05-19T22:01:30.499162581+02:00 stderr F [Tue May 19 22:01:30.499149 2020] [so:warn] [pid 8] AH01574: module dav_module is already loaded, skipping
2020-05-19T22:01:30.499181334+02:00 stderr F [Tue May 19 22:01:30.499171 2020] [so:warn] [pid 8] AH01574: module dav_fs_module is already loaded, skipping
2020-05-19T22:01:30.500983929+02:00 stderr F [Tue May 19 22:01:30.500619 2020] [so:warn] [pid 8] AH01574: module mpm_prefork_module is already loaded, skipping
2020-05-19T22:01:30.504247745+02:00 stderr F [Tue May 19 22:01:30.504208 2020] [so:warn] [pid 8] AH01574: module ssl_module is already loaded, skipping
2020-05-19T22:01:30.504293710+02:00 stderr F [Tue May 19 22:01:30.504272 2020] [so:warn] [pid 8] AH01574: module systemd_module is already loaded, skipping
2020-05-19T22:01:30.504417905+02:00 stderr F [Tue May 19 22:01:30.504396 2020] [so:warn] [pid 8] AH01574: module cgi_module is already loaded, skipping
2020-05-19T22:01:30.537590475+02:00 stderr F [Tue May 19 22:01:30.537544 2020] [so:warn] [pid 8] AH01574: module wsgi_module is already loaded, skipping
2020-05-19T22:01:30.547094328+02:00 stderr F (98)Address already in use: AH00072: make_sock: could not bind to address 192.168.254.100:35357
2020-05-19T22:01:30.547094328+02:00 stderr F no listening sockets available, shutting down
2020-05-19T22:01:30.547094328+02:00 stderr F AH00015: Unable to open logs

/var/log/containers/keystone/keystone.log

2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application [req-8709dd4e-fee1-422c-99e6-9e12a12d3091 - - - - -] An unexpected error prevented the server from fulfilling your request.: KeysNotFound: An unexpected error prevented the server from fulfilling your request.
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application Traceback (most recent call last):
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application File "/usr/lib/python2.7/site-packages/flask_restful/__init__.py", line 270, in error_router
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application return self.handle_error(e)
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application File "/usr/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application rv = self.dispatch_request()
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application File "/usr/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application return self.view_functions[rule.endpoint](**req.view_args)
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application File "/usr/lib/python2.7/site-packages/flask_restful/__init__.py", line 480, in wrapper
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application resp = resource(*args, **kwargs)
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application File "/usr/lib/python2.7/site-packages/flask/views.py", line 88, in view
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application return self.dispatch_request(*args, **kwargs)
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application File "/usr/lib/python2.7/site-packages/flask_restful/__init__.py", line 595, in dispatch_request
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application resp = meth(*args, **kwargs)
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application File "/usr/lib/python2.7/site-packages/keystone/server/flask/common.py", line 1066, in wrapper
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application return f(*args, **kwargs)
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application File "/usr/lib/python2.7/site-packages/keystone/api/auth.py", line 315, in post
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application token = authentication.authenticate_for_token(auth_data)
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application File "/usr/lib/python2.7/site-packages/keystone/api/_shared/authentication.py", line 229, in authenticate_for_token
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application app_cred_id=app_cred_id, parent_audit_id=token_audit_id)
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application File "/usr/lib/python2.7/site-packages/keystone/common/manager.py", line 116, in wrapped
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs)
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application File "/usr/lib/python2.7/site-packages/keystone/token/provider.py", line 272, in issue_token
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application token_id, issued_at = self.driver.generate_id_and_issued_at(token)
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application File "/usr/lib/python2.7/site-packages/keystone/token/providers/fernet/core.py", line 86, in generate_id_and_issued_at
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application app_cred_id=token.application_credential_id
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application File "/usr/lib/python2.7/site-packages/keystone/token/token_formatters.py", line 153, in create_token
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application token = self.pack(serialized_payload)
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application File "/usr/lib/python2.7/site-packages/keystone/token/token_formatters.py", line 79, in pack
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application return self.crypto.encrypt(payload).rstrip(b'=').decode('utf-8')
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application File "/usr/lib/python2.7/site-packages/keystone/token/token_formatters.py", line 66, in crypto
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application raise exception.KeysNotFound()
2020-05-19 22:53:00.026 224 ERROR keystone.server.flask.application KeysNotFound: An unexpected error prevented the server from fulfilling your request.

[root@undercloud stdouts]# tail rabbitmq.log
2020-05-19T22:57:26.393172984+02:00 stderr F ++ : /var/log/kolla/rabbitmq
2020-05-19T22:57:26.393302877+02:00 stderr F ++ [[ -n '' ]]
2020-05-19T22:57:26.393348665+02:00 stderr F ++ [[ ! -d /var/log/kolla/rabbitmq ]]
2020-05-19T22:57:26.394282606+02:00 stderr F +++ stat -c %a /var/log/kolla/rabbitmq
2020-05-19T22:57:26.396314178+02:00 stderr F ++ [[ 2755 != \7\5\5 ]]
2020-05-19T22:57:26.396314178+02:00 stderr F ++ chmod 755 /var/log/kolla/rabbitmq
2020-05-19T22:57:26.397201207+02:00 stderr F + echo 'Running command: '\''/usr/lib/rabbitmq/bin/rabbitmq-server'\'''
2020-05-19T22:57:26.397232044+02:00 stdout F Running command: '/usr/lib/rabbitmq/bin/rabbitmq-server'
2020-05-19T22:57:26.397267329+02:00 stderr F + exec /usr/lib/rabbitmq/bin/rabbitmq-server
2020-05-19T22:57:27.719392919+02:00 stdout F ERROR: node with name "rabbit" already running on "undercloud"

[Harald Jensås (harald-jensas)]

A reboot, and re-run of 'openstack undercloud upgrade' solved the problem.

[Marios Andreou (marios-b)]

Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo

[Launchpad Janitor (janitor)]

[Expired for tripleo because there has been no activity for 60 days.]