Intermittent functional test failure

[Frode Nordahl]

DEBUG:runner:call ['/tmp/bundletester-ryUT53/keystone-ldap/tests/gate-basic-bionic-stein'] (cwd: /tmp/bundletester-ryUT53/keystone-ldap)
DEBUG:runner:2019-08-22 09:44:00,560 __init__ INFO: OpenStackAmuletDeployment: init
DEBUG:runner:2019-08-22 09:44:00,560 _add_services INFO: OpenStackAmuletDeployment: adding services
DEBUG:runner:2019-08-22 09:44:00,560 _determine_branch_locations INFO: OpenStackAmuletDeployment: determine branch locations
DEBUG:runner:2019-08-22 09:44:02 Starting deployment of auto-osci-sv06:admin/auto-osci-sv06
DEBUG:runner:2019-08-22 09:44:02 Deploying applications...
DEBUG:runner:2019-08-22 09:44:02 Deploying application keystone using cs:~openstack-charmers-next/keystone-453
DEBUG:runner:2019-08-22 09:44:05 Deploying application keystone-ldap using /tmp/charm9BOIVL/bionic/keystone-ldap
DEBUG:runner:2019-08-22 09:44:08 Deploying application ldap-server using cs:~openstack-charmers/ldap-test-fixture-3
DEBUG:runner:2019-08-22 09:44:11 Deploying application percona-cluster using cs:~openstack-charmers-next/percona-cluster-353
DEBUG:runner:2019-08-22 09:44:18 Config specifies num units for subordinate: keystone-ldap
DEBUG:runner:2019-08-22 09:50:19 Adding relations...
DEBUG:runner:2019-08-22 09:50:20 Adding relation keystone:shared-db <-> percona-cluster:shared-db
DEBUG:runner:2019-08-22 09:50:20 Adding relation keystone:domain-backend <-> keystone-ldap:domain-backend
DEBUG:runner:2019-08-22 09:53:31 Deployment complete in 569.72 seconds
DEBUG:runner:2019-08-22 09:54:01,502 _configure_services INFO: OpenStackAmuletDeployment: configure services
DEBUG:runner:2019-08-22 09:54:02,453 __init__ INFO: Waiting on extended status checks...
DEBUG:runner:2019-08-22 09:54:02,454 _auto_wait_for_status INFO: Waiting for extended status on units for 5400s...
DEBUG:runner:2019-08-22 09:54:02,454 _auto_wait_for_status DEBUG: Default extended status wait match: contains READY (case-insensitive)
DEBUG:runner:2019-08-22 09:54:02,457 _auto_wait_for_status DEBUG: Excluding services from extended status match: ['mysql', 'mongodb']
DEBUG:runner:2019-08-22 09:54:02,457 _auto_wait_for_status DEBUG: Waiting up to 5400s for extended status on services: ['keystone-ldap', 'keystone', 'ldap-server', 'percona-cluster']
DEBUG:runner:2019-08-22 09:54:08,342 _auto_wait_for_status INFO: OK
DEBUG:runner:2019-08-22 09:55:22,460 get_default_keystone_session DEBUG: Authenticating keystone admin...
DEBUG:runner:Traceback (most recent call last):
DEBUG:runner: File "/tmp/bundletester-ryUT53/keystone-ldap/tests/gate-basic-bionic-stein", line 24, in <module>
DEBUG:runner: deployment.run_tests()
DEBUG:runner: File "/tmp/tmp.h1wclcDSWj/func27/local/lib/python2.7/site-packages/charmhelpers/contrib/amulet/deployment.py", line 99, in run_tests
DEBUG:runner: getattr(self, test)()
DEBUG:runner: File "/tmp/bundletester-ryUT53/keystone-ldap/tests/basic_deployment.py", line 156, in test_100_keystone_ldap_users
DEBUG:runner: 'john doe', 'userdomain')
DEBUG:runner: File "/tmp/bundletester-ryUT53/keystone-ldap/tests/basic_deployment.py", line 135, in find_keystone_v3_user
DEBUG:runner: domain=client.domains.find(name=domain).id
DEBUG:runner: File "/tmp/tmp.h1wclcDSWj/func27/local/lib/python2.7/site-packages/keystoneclient/base.py", line 86, in func
DEBUG:runner: return f(*args, **new_kwargs)
DEBUG:runner: File "/tmp/tmp.h1wclcDSWj/func27/local/lib/python2.7/site-packages/keystoneclient/base.py", line 504, in find
DEBUG:runner: raise ksa_exceptions.NotFound(404, msg)
DEBUG:runner:keystoneauth1.exceptions.http.NotFound: 404 (HTTP 404)
DEBUG:runner:Exit Code: 1

Model Controller Cloud/Region Version SLA Timestamp
auto-osci-sv06 auto-osci-sv06 serverstack/serverstack 2.6.7 unsupported 09:55:34Z

App Version Status Scale Charm Store Rev OS Notes
keystone 15.0.0 active 1 keystone jujucharms 453 ubuntu
keystone-ldap 15.0.0 active 1 keystone-ldap local 2 ubuntu
ldap-server active 1 ldap-test-fixture jujucharms 3 ubuntu
percona-cluster 5.7.20-29.24 active 1 percona-cluster jujucharms 353 ubuntu

Unit Workload Agent Machine Public address Ports Message
keystone/2* active executing 6 172.17.106.11 5000/tcp Unit is ready
  keystone-ldap/2* active idle 172.17.106.11 Unit is ready
ldap-server/2* active idle 7 172.17.106.12 Unit is ready
percona-cluster/2* active idle 8 172.17.106.6 3306/tcp Unit is ready

Machine State DNS Inst id Series AZ Message
6 started 172.17.106.11 514901b4-e2b6-4475-a715-b8f9648918b8 bionic nova ACTIVE
7 started 172.17.106.12 c7318105-685f-41f6-a51a-c877170b602e bionic nova ACTIVE
8 started 172.17.106.6 82d114a8-d0f9-403d-b6b2-93871e261eea bionic nova ACTIVE

Nothing obvious in charm, keystone or Apache logs.

[Ryan Beisner (1chb1n)]

Example:

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-keystone-ldap/677920/1/3645/index.html

[Alex Kavanagh (ajkavanagh)]

Setting to incomplete as its an intermittent failure; please re-check in 60 days and if still a problem reset to new.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Keystone LDAP integration because there has been no activity for 60 days.]

[Corey Bryant (corey.bryant)]

Not the same failures as those reported in the original bug description, but there still seem to be intermittent failures with these functional tests. Please see attached doc. Tests failed in the gate in https://review.opendev.org/c/openstack/charm-keystone-ldap/+/873802 but I was able to run successfully on my own deploy.

[Corey Bryant (corey.bryant)]

The issue I mentioned here should be fixed by: https://github.com/openstack-charmers/zaza-openstack-tests/pull/1023