devstack

Bad response code while validating token: 502

Bug #1681866 reported by Jordan Pittier on 2017-04-11

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	devstack	Expired	Undecided	Unassigned

Bug Description

Found this while investigating a gate failure [1].

Tempest logs say "2017-04-11 10:07:02,765 23082 INFO [tempest.lib.common.rest_client] Request (TestSecurityGroupsBasicOps:_run_cleanups): 503 DELETE https://198.72.124.138:8774/v2.1/servers/f736a878-2ac4-4c37-b6a8-e5cd8df5a7fd 0.018s"

That 503 looks suspicious. So I go to the nova-api logs. Which gives

2017-04-11 10:07:02.762 32191 ERROR keystonemiddleware.auth_token [...] Bad response code while validating token: 502
2017-04-11 10:07:02.763 32191 WARNING keystonemiddleware.auth_token [...] Identity response: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request <em><a href="/identity_admin/v3/auth/tokens">GET /identity_admin/v3/auth/tokens</a></em>.<p>
Reason: <strong>Error reading from remote server</strong></p></p>
<hr>
<address>Apache/2.4.18 (Ubuntu) Server at 198.72.124.138 Port 443</address>
</body></html>

2017-04-11 10:07:02.763 32191 CRITICAL keystonemiddleware.auth_token [...] Unable to validate token: Failed to fetch token data from identity server

So Apache is complaining, some network connection issue, related to proxy-ing. So I open "logs/apache/tls-proxy_error.txt.gz" and find

[Tue Apr 11 10:07:02.761420 2017] [proxy_http:error] [pid 7136:tid 140090189690624] (20014)Internal error (specific information not available): [client 198.72.124.138:38722] [frontend 198.72.124.138:443] AH01102: error reading status line from remote server 198.72.124.138:80
[Tue Apr 11 10:07:02.761454 2017] [proxy:error] [pid 7136:tid 140090189690624] [client 198.72.124.138:38722] [frontend 198.72.124.138:443] AH00898: Error reading from remote server returned by /identity_admin/v3/auth/tokens

Interesting. Google says that adding "proxy-initial-not-pooled" to the apache2 vhost config could help.

Anyway, a good elasticsearch query for this is

message:"Bad response code while validating token: 502"

8 hits, no worries.

[1] : http://logs.openstack.org/03/455303/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/aa8c7fd/console.html

Ken'ichi Ohmichi (oomichi) on 2017-04-26

Changed in tempest:
status:	New → Confirmed
importance:	Undecided → Medium