Ocata -> Pike: overcloud upgrade returns 500 error at the end (race condition?)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Invalid
|
High
|
Unassigned |
Bug Description
Dear Stackers,
While wanting to upgrade our lab ocata to pike, we got a weird situation.
Used command:
openstack overcloud deploy \
--templates $TEMPLATES \
-e ./upgrade-pike.yaml \
-e ./openstack-
$ENVIRONMENTS
The "upgrade-pike.yaml" contains the commands to be executed in order to update the source lists on the nodes, as per http://
The other variables are set and points to the correct directories/files.
All seemed to work, but in the end we got that error:
2017-08-24 09:50:51Z [overcloud]: UPDATE_COMPLETE Stack UPDATE completed successfully
Stack overcloud UPDATE_COMPLETE
Internal Server Error (HTTP 500)
Yep, "COMPLETE" but right after that we got a 500 error, meaning, in the end, "failed".
The current state: horizon doesn't allow any login (returns a 500), but we apparently can do calls to the API directly using the `openstack` command.
After digging a bit more and deeper, I found this weird error in /var/log/
2017-08-25 05:36:15.988 482029 DEBUG keystone.
Searching for the error message, I stumbled on many occurrences, for older version, and they apparently were solved in the meanwhile, like:
- https:/
- https:/
- https:/
I'm not 100% sure this is the realy issue, as it's a DEBUG message and not an ERROR, and not 100% sure keystone is broken as we actually can call the API (note: I didn't test all the capabilities - only `openstack server list`).
Has anyone already hit that situation?
Thanks in advance.
Best regards,
C.
Changed in tripleo: | |
status: | New → Triaged |
importance: | Undecided → High |
milestone: | none → pike-rc2 |
tags: | added: upgrade |
Changed in tripleo: | |
milestone: | pike-rc2 → queens-1 |
Changed in tripleo: | |
milestone: | queens-1 → queens-2 |
Changed in tripleo: | |
milestone: | queens-2 → queens-3 |
Changed in tripleo: | |
milestone: | queens-3 → queens-rc1 |
Changed in tripleo: | |
milestone: | queens-rc1 → rocky-1 |
Changed in tripleo: | |
milestone: | rocky-1 → rocky-2 |
Changed in tripleo: | |
milestone: | rocky-2 → rocky-3 |
Changed in tripleo: | |
milestone: | rocky-3 → rocky-rc1 |
Changed in tripleo: | |
milestone: | rocky-rc1 → stein-1 |
Changed in tripleo: | |
milestone: | stein-1 → stein-2 |
Changed in tripleo: | |
milestone: | stein-2 → stein-3 |
Changed in tripleo: | |
status: | Triaged → Invalid |
Update: might be unrelated to keystone, more likely some race condition between something restarted on the overcloud services and some query done by the deploy process (it wants to show the keystone endpoint at the end, right?)