tripleo-ci-centos-7-scenario001-multinode-oooq-container is failing on deploying the overcloud - master release only

[Ronelle Landy]

tripleo-ci-centos-7-scenario001-multinode-oooq-container is failing has started failing consistently on deploying the overcloud with the following failure logs:

http://logs.openstack.org/07/472607/133/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/443d9e8/job-output.txt.gz

http://logs.openstack.org/07/472607/133/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/443d9e8/logs/undercloud/home/zuul/overcloud

http://logs.openstack.org/07/472607/133/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/443d9e8/logs/undercloud/home/zuul/failed_deployment_list.log.txt.gz

This test was moved to non-voting due to tempest errors.

[Ronelle Landy (rlandy)]

Pradeep Kilambi is investigating ...

<pradk> hmm seems like the ipmi agent is not starting
<pradk> http://logs.openstack.org/79/517079/4/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/1f8d5cd/logs/subnode-2/var/log/extra/docker/containers/ceilometer_agent_ipmi/stdout.log.txt.gz

[Alex Schultz (alex-schultz)]

Looks to be failing trying to run gnocchi db sync

http://logs.openstack.org/31/522231/1/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/013cc53/logs/postci.txt.gz

            "Error running ['docker', 'run', '--name', 'gnocchi_db_sync', '--label', 'config_id=tripleo_step4', '--label', 'container_name=gnocchi_db_sync', '--label', 'managed_by=paunch', '--label', 'config_data={\"command\": \"/usr/bin/bootstrap_host_exec gnocchi_api su gnocchi -s /bin/bash -c \\'/usr/bin/gnocchi-upgrade --sacks-number=128\\'\", \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/var/lib/config-data/gnocchi/etc/my.cnf.d/tripleo.cnf:/etc/my.cnf.d/tripleo.cnf:ro\", \"/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro\", \"/var/log/containers/gnocchi:/var/log/gnocchi\", \"/var/log/containers/httpd/gnocchi-api:/var/log/httpd\", \"/etc/ceph:/etc/ceph:ro\"], \"image\": \"192.168.24.1:8787/tripleomaster/centos-binary-gnocchi-api:3b85715c6488a692fcdf4e4c6a589d77c60eb5a6_a3792c8d\", \"detach\": false, \"net\": \"host\", \"privileged\": false}', '--net=host', '--privileged=false', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/var/lib/config-data/gnocchi/etc/my.cnf.d/tripleo.cnf:/etc/my.cnf.d/tripleo.cnf:ro', '--volume=/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro', '--volume=/var/log/containers/gnocchi:/var/log/gnocchi', '--volume=/var/log/containers/httpd/gnocchi-api:/var/log/httpd', '--volume=/etc/ceph:/etc/ceph:ro', '192.168.24.1:8787/tripleomaster/centos-binary-gnocchi-api:3b85715c6488a692fcdf4e4c6a589d77c60eb5a6_a3792c8d', '/usr/bin/bootstrap_host_exec', 'gnocchi_api', 'su', 'gnocchi', '-s', '/bin/bash', '-c', \"'/usr/bin/gnocchi-upgrade\", \"--sacks-number=128'\"]. [1]",

[Alex Schultz (alex-schultz)]

http://logs.openstack.org/31/522231/1/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/013cc53/logs/subnode-2/var/log/containers/gnocchi/gnocchi-upgrade.log.txt.gz

2017-11-27 17:16:25,819 [14] CRITICAL root: Traceback (most recent call last):
  File "/usr/bin/gnocchi-upgrade", line 10, in <module>
    sys.exit(upgrade())
  File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 66, in upgrade
    s = storage.get_driver(conf)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 163, in get_driver
    conf.storage, incoming, coord)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 48, in __init__
    self.rados, self.ioctx = ceph.create_rados_connection(conf)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/common/ceph.py", line 68, in create_rados_connection
    conn.connect()
  File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969)
PermissionDeniedError: error connecting to the cluster

[Pradeep Kilambi (pkilambi)]

This seems to be the same issue as https://bugs.launchpad.net/tripleo/+bug/1734134 .. looking at the traceback

[Ronelle Landy (rlandy)]

Following Pradeep's comment above, marking this bug as a duplicate of https://bugs.launchpad.net/tripleo/+bug/1734134.