keystone cli tests fail in grizzly

[Xavier Queralt]

Tempest tests fail with the following errors in grizzly:

http://paste.openstack.org/show/44510/

Two keystone tests fail while checking the output of the command. From what I can see, the last time tempest tests passed on grizzly was on 17th of August (http://logs.openstack.org/00/42400/1/check/gate-tempest-devstack-vm-full/a7b5629/console.html.gz)

Here are a couple of tempest reviews with the problem:

http://logs.openstack.org/88/42588/1/check/gate-tempest-devstack-vm-full/a945ba5/console.html

http://logs.openstack.org/57/41657/3/check/gate-tempest-devstack-vm-full/f7d8f28/console.html

[Alan Pevec (apevec)]

Featured in stable-maint thread: http://lists.openstack.org/pipermail/openstack-stable-maint/2013-August/001188.html

[Pavel Sedlák (psedlak)]

With stable/grizzly branch of everything where it is available (keystone, nova...) and master branch of devstack and tempest, clients etc, I was unable to reproduce on fedora19.
Neither with just master python-keystoneclient and tempest launched against older grizzly installation.

In my setup nova failed to start because i have a lot of python-keystoneclient version conflicts there:
- nova throws
   pkg_resources.VersionConflict: (python-keystoneclient 0.2.5 (/usr/lib/python2.7/site-packages), Requirement.parse(\'python-keystoneclient>=0.3.0\')
   from nova-rootwrap, require(\'nova...)
- after doing doing pip uninstall p-key-client && pip install -e /opt/stack/new/p-key-client
   again something from /usr/bin/nova-rootwrap requires
   pkg_resources.DistributionNotFound: python-keystoneclient>=0.2,<0.3
   require(\'nova==2013.1.4.a1.g37e3b55\')

After I've checked the console logs from those example failed runs, I see that keystonclient was re-installed multiple times 2.5 vs 3.1 and back, so when it's maybe not related to this bug, we are still back at issue with inconsistent dependencies.

As I was not yet able to reproduce this at all, I've improved those assertTrue tests little bit https://review.openstack.org/#/c/43350/ so we can get better idea what's happening there.

[Dolph Mathews (dolph)]

Thanks for the analysis Pavel!

[Pavel Sedlák (psedlak)]

And here - https://jenkins01.openstack.org/job/gate-tempest-devstack-vm-full/4193/console - is the reason:

> FAIL: cli.simple_read_only.test_keystone.SimpleReadOnlyKeystoneClientTest.test_admin_catalog_list
> ----------------------------------------------------------------------
> AssertionError: Invalid beginning of service block: ERROR:root:Could not find any typelib for GnomeKeyring

> FAIL: cli.simple_read_only.test_keystone.SimpleReadOnlyKeystoneClientTest.test_admin_help
> ----------------------------------------------------------------------
> self.assertFirstLineStartsWith(lines, 'usage: keystone')
> AssertionError: Beginning of first line has invalid content: ['ERROR:root:Could not find any typelib for GnomeKeyring' ...

So it's the bug 1193164.

[Pavel Sedlák (psedlak)]

To make stable/grizzly 'contributable' again, I proposed skip in https://review.openstack.org/#/c/43424/

Proper solution could be determined in Bug 1193164.

[Christopher Yeoh (cyeoh-0)]

Also seeing a very similar failure in master http://logs.openstack.org/31/40331/3/gate/gate-tempest-devstack-vm-postgres-full/861438d/testr_results.html.gz

[Dolph Mathews (dolph)]

Christopher Yeoh: I think that might be a discrete issue? There are at least two reports of that specific failure, one of which is bug 1217265

[Dolph Mathews (dolph)]

Linked the wrong bug in comment #7 (although that might be somehow related too..). I meant to link to bug 1217159