Fuel for OpenStack

  1. Bug #1588329
  2. Comment #6

Comment 6 for bug 1588329

Revision history for this message
Alex Schultz (alex-schultz) wrote : Re: [9.0] Packaging CI job failed on a heat-primary task

To follow up after looking in the haproxy logs of the last snapshot, the keystone token request took excessively long causing the puppet task to timeout since we have a 20 second timeout.

This call took ~18 seconds.

2016-06-16T01:00:08.810967+00:00 info: 10.109.1.2:60911 [16/Jun/2016:00:59:49.987] keystone-1 keystone-1/node-3 0/0/3/18867/18870 201 8194 - - ---- 57/0/0/0/0 0/0 "POST /v3/auth/tokens HTTP/1.1"

This call took ~36 seconds.
2016-06-16T01:00:49.798832+00:00 info: 10.109.1.2:32949 [16/Jun/2016:01:00:13.134] keystone-1 keystone-1/node-2 4/0/6/36694/36704 201 8194 - - ---- 55/1/1/1/0 0/0 "POST /v3/auth/tokens HTTP/1.1"

This call took ~13 seconds
2016-06-16T01:00:49.908022+00:00 info: 10.109.1.2:33215 [16/Jun/2016:01:00:36.158] keystone-1 keystone-1/node-2 0/0/1/13789/13793 201 8194 - - ---- 55/0/0/0/0 0/0 "POST /v3/auth/tokens HTTP/1.1"

So because all the calls took excessively long at the time, the user creation fails. This probably points to the keystone (or CI) servers being overloaded at the time of the heat task. We could increase the timeout of the user creation from 20 seconds to 60 seconds as a possible work around.