tempest-full job failing in stable/pike with 404 from keystone during tempest verify-config
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tempest |
Fix Released
|
Critical
|
Andreas Jaeger |
Bug Description
Been seeing this in pike patches today:
2017-12-22 16:23:08.888450 | controller | 2017-12-22 16:23:08.888 | venv runtests: PYTHONHASHSEED=
2017-12-22 16:23:08.888509 | controller | 2017-12-22 16:23:08.888 | venv runtests: commands[0] | tempest verify-config -uro ../../.
2017-12-22 16:23:09.523336 | controller | 2017-12-22 16:23:09.522 | Traceback (most recent call last):
2017-12-22 16:23:09.523425 | controller | 2017-12-22 16:23:09.522 | File "/opt/stack/
2017-12-22 16:23:09.523470 | controller | 2017-12-22 16:23:09.522 | main(parsed_args)
2017-12-22 16:23:09.523523 | controller | 2017-12-22 16:23:09.522 | File "/opt/stack/
2017-12-22 16:23:09.523569 | controller | 2017-12-22 16:23:09.522 | 'verify_
2017-12-22 16:23:09.523624 | controller | 2017-12-22 16:23:09.522 | File "/opt/stack/
2017-12-22 16:23:09.523663 | controller | 2017-12-22 16:23:09.522 | **get_dynamic_
2017-12-22 16:23:09.523715 | controller | 2017-12-22 16:23:09.522 | File "/opt/stack/
2017-12-22 16:23:09.523753 | controller | 2017-12-22 16:23:09.522 | fill_in=True, identity_
2017-12-22 16:23:09.523807 | controller | 2017-12-22 16:23:09.522 | File "/opt/stack/
2017-12-22 16:23:09.523844 | controller | 2017-12-22 16:23:09.522 | identity_
2017-12-22 16:23:09.523892 | controller | 2017-12-22 16:23:09.522 | File "/opt/stack/
2017-12-22 16:23:09.523920 | controller | 2017-12-22 16:23:09.522 | **params)
2017-12-22 16:23:09.523964 | controller | 2017-12-22 16:23:09.522 | File "/opt/stack/
2017-12-22 16:23:09.523999 | controller | 2017-12-22 16:23:09.523 | creds = auth_provider.
2017-12-22 16:23:09.524043 | controller | 2017-12-22 16:23:09.523 | File "/opt/stack/
2017-12-22 16:23:09.524075 | controller | 2017-12-22 16:23:09.523 | auth_data = self.get_auth()
2017-12-22 16:23:09.524117 | controller | 2017-12-22 16:23:09.523 | File "/opt/stack/
2017-12-22 16:23:09.524145 | controller | 2017-12-22 16:23:09.523 | self.set_auth()
2017-12-22 16:23:09.524187 | controller | 2017-12-22 16:23:09.523 | File "/opt/stack/
2017-12-22 16:23:09.524219 | controller | 2017-12-22 16:23:09.523 | self.cache = self._get_auth()
2017-12-22 16:23:09.524261 | controller | 2017-12-22 16:23:09.523 | File "/opt/stack/
2017-12-22 16:23:09.524297 | controller | 2017-12-22 16:23:09.523 | token, auth_data = auth_func(
2017-12-22 16:23:09.524347 | controller | 2017-12-22 16:23:09.523 | File "/opt/stack/
2017-12-22 16:23:09.524389 | controller | 2017-12-22 16:23:09.523 | body = self.auth(**kwargs)
2017-12-22 16:23:09.524440 | controller | 2017-12-22 16:23:09.523 | File "/opt/stack/
2017-12-22 16:23:09.524477 | controller | 2017-12-22 16:23:09.523 | resp, body = self.post(
2017-12-22 16:23:09.524523 | controller | 2017-12-22 16:23:09.523 | File "/opt/stack/
2017-12-22 16:23:09.524566 | controller | 2017-12-22 16:23:09.523 | return self.request(
2017-12-22 16:23:09.524616 | controller | 2017-12-22 16:23:09.523 | File "/opt/stack/
2017-12-22 16:23:09.524653 | controller | 2017-12-22 16:23:09.523 | 'Unexpected status code {0}'.format(
2017-12-22 16:23:09.524686 | controller | 2017-12-22 16:23:09.523 | IdentityError: Got identity error
2017-12-22 16:23:09.524718 | controller | 2017-12-22 16:23:09.523 | Details: Unexpected status code 404
2017-12-22 16:23:09.524793 | controller | 2017-12-22 16:23:09.524 | Running config verification...
2017-12-22 16:23:09.553728 | controller | 2017-12-22 16:23:09.553 | ERROR: InvocationError: '/opt/stack/
Looks like the actual failure is trying to create a token:
2017-12-22 16:23:09.520 27885 INFO tempest.
2017-12-22 16:23:09.521 27885 DEBUG tempest.
Body: <omitted>
Response - Headers: {u'date': 'Fri, 22 Dec 2017 16:23:09 GMT', 'status': '404', u'server': 'Apache/2.4.18 (Ubuntu)', u'content-length': '300', 'content-location': 'http://
Body: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /identity/
<hr>
<address>
</body></html>
_log_request_full /opt/stack/
Because the log path has changed, the controller/logs/ aren't index in logstash so I can only fingerprint this based on the failure in the console log.
Changed in tempest: | |
importance: | Undecided → Critical |
Changed in tempest: | |
assignee: | nobody → Armando Migliaccio (armando-migliaccio) |
status: | Confirmed → In Progress |
Changed in tempest: | |
assignee: | Armando Migliaccio (armando-migliaccio) → Andreas Jaeger (jaegerandi) |
Changed in tempest: | |
status: | In Progress → Fix Released |
It is reproducible.
http:// logs.openstack. org/87/ 529987/ 1/check/ tempest- full/9a2fa46/ job-output. txt.gz
2017-12-26 03:51:06.686366 | controller | 2017-12-26 03:51:06.686 | venv runtests: PYTHONHASHSEED= '2489918815' ./tmp/tmp. 9bqD2wFOA4 tempest/ tempest/ cmd/verify_ tempest_ config. py", line 486, in take_action tempest/ tempest/ cmd/verify_ tempest_ config. py", line 450, in main tempest_ config' , network_ resources= net_resources) tempest/ tempest/ common/ credentials_ factory. py", line 141, in get_credentials _provider provider_ params( identity_ version) ) tempest/ tempest/ common/ credentials_ factory. py", line 73, in get_dynamic_ provider_ params version= identity_ version) tempest/ tempest/ common/ credentials_ factory. py", line 254, in get_configured_ admin_credentia ls version= identity_ version, **params) tempest/ tempest/ common/ credentials_ factory. py", line 298, in get_credentials tempest/ tempest/ lib/auth. py", line 648, in get_credentials fill_credential s() tempest/ tempest/ lib/auth. py", line 124, in fill_credentials tempest/ tempest/ lib/auth. py", line 150, in get_auth tempest/ tempest/ lib/auth. py", line 159, in set_auth
2017-12-26 03:51:06.686415 | controller | 2017-12-26 03:51:06.686 | venv runtests: commands[0] | tempest verify-config -uro ../../.
2017-12-26 03:51:07.355947 | controller | 2017-12-26 03:51:07.355 | Traceback (most recent call last):
2017-12-26 03:51:07.356057 | controller | 2017-12-26 03:51:07.355 | File "/opt/stack/
2017-12-26 03:51:07.356133 | controller | 2017-12-26 03:51:07.355 | main(parsed_args)
2017-12-26 03:51:07.356211 | controller | 2017-12-26 03:51:07.356 | File "/opt/stack/
2017-12-26 03:51:07.356272 | controller | 2017-12-26 03:51:07.356 | 'verify_
2017-12-26 03:51:07.356362 | controller | 2017-12-26 03:51:07.356 | File "/opt/stack/
2017-12-26 03:51:07.356420 | controller | 2017-12-26 03:51:07.356 | **get_dynamic_
2017-12-26 03:51:07.356492 | controller | 2017-12-26 03:51:07.356 | File "/opt/stack/
2017-12-26 03:51:07.356542 | controller | 2017-12-26 03:51:07.356 | fill_in=True, identity_
2017-12-26 03:51:07.356623 | controller | 2017-12-26 03:51:07.356 | File "/opt/stack/
2017-12-26 03:51:07.356687 | controller | 2017-12-26 03:51:07.356 | identity_
2017-12-26 03:51:07.356760 | controller | 2017-12-26 03:51:07.356 | File "/opt/stack/
2017-12-26 03:51:07.356795 | controller | 2017-12-26 03:51:07.356 | **params)
2017-12-26 03:51:07.356853 | controller | 2017-12-26 03:51:07.356 | File "/opt/stack/
2017-12-26 03:51:07.356897 | controller | 2017-12-26 03:51:07.356 | creds = auth_provider.
2017-12-26 03:51:07.356955 | controller | 2017-12-26 03:51:07.356 | File "/opt/stack/
2017-12-26 03:51:07.356995 | controller | 2017-12-26 03:51:07.356 | auth_data = self.get_auth()
2017-12-26 03:51:07.357049 | controller | 2017-12-26 03:51:07.356 | File "/opt/stack/
2017-12-26 03:51:07.357083 | controller | 2017-12-26 03:51:07.356 | self.set_auth()
2017-12-26 03:51:07.357137 | controller | 2017-12-26 03:51:07.356 | File "/opt/stack/
2017-12-26 03:51:07.357178 | controller | 2017-12-26 03:51:07.356 | self.cache = self._get_auth()
2017-12-26 03:51:07.357259 | controller | 2017-12-26 03:51:07.356 | File "/opt/stack/tem...