keystone reports 404 User Not Found during grenade tests

[Lance Bragstad]

During the keystone meeting today [0], clarkb brought this to our attention.

At some point during grenade tests [1], keystone will report 404 User Not Found [2].

Opening this for primarily for investigative purposes until we figure out what is going on.

[0] http://eavesdrop.openstack.org/meetings/keystone/2017/keystone.2017-05-30-18.00.log.html#l-122
[1] http://logs.openstack.org/36/458636/16/gate/gate-grenade-dsvm-neutron-ubuntu-xenial/5f07eaa/logs/tempest.txt.gz?level=DEBUG#_2017-05-23_19_00_51_098
[2] http://logs.openstack.org/36/458636/16/gate/gate-grenade-dsvm-neutron-ubuntu-xenial/5f07eaa/logs/apache/keystone.txt.gz?level=WARNING#_2017-05-23_19_01_18_408

[Lance Bragstad (lbragstad)]

Using the following logstash query [0], and there is a lot of service account names not found. For example:

  Could not find user: swift
  Could not find user: glance-swift
  Could not find user: neutron

[0] http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Could%20not%20find%20user:%5C%22%20AND%20tags:%5C%22keystone.txt%5C%22

[Colleen Murphy (krinkle)]

The logs for this example seem to have been rotated away :(

It's hard to say that this is actually a problem with keystone. The way openstackclient works, if you call for example `openstack user show nova`, it doesn't know whether it has a name or an ID, so it tries both, e.g. GET /users/nova and then GET /users?name=nova. Keystone logs a warning for the first one, which is the right thing to do, but it does unfortunately clutter up the logs. You can see an example here:

http://logs.openstack.org/64/371564/4/gate/gate-grenade-dsvm-neutron-ubuntu-xenial/7e154f3/logs/apache/keystone.txt.gz#_2017-08-08_06_36_12_737

where it logs a warning but then finds the nova user a few lines down.

[Lance Bragstad (lbragstad)]

The logstash query isn't exactly helpful. I assume because of all the false positives. Maybe we can loop Clark back in to see if I missed something in the original report.

[Lance Bragstad (lbragstad)]

Closing this based on the discussion in IRC [0]. Let's reopen if this pops up again.

[0] http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2017-08-08.log.html#t2017-08-08T21:21:13