docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container

[wes hayutin]

According to http://cistatus.tripleo.org/ -> gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container has been failing 100% of the time since 08-08-2017.

50% of the time the overcloud fails to deploy, the other 50% the job times out.

Creating the bug w/ an alert and starting to debug

[wes hayutin (weshayutin)]

http://logs.openstack.org/23/491923/1/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/435f1fe/logs/undercloud/home/jenkins/failed_deployment_list.log.txt.gz

http://logs.openstack.org/29/491929/2/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/4d0bd08/logs/undercloud/home/jenkins/failed_deployment_list.log.txt.gz

 "INFO [alembic.runtime.migration] Running upgrade aabe895bbd4d -> c0125080d572, policy library",
            "Error running ['docker', 'run', '--name', 'gnocchi_db_sync', '--label', 'config_id=tripleo_step3', '--label', 'container_name=gnocchi_db_sync', '--label', 'managed_by=paunch', '--label', 'config_data={\"image\": \"192.168.24.1:8787/tripleoupstream/centos-binary-gnocchi-api:latest\", \"environment\": [\"TRIPLEO_CONFIG_HASH=f264b1307217a5596bbcba581fee1833\"], \"command\": \"/usr/bin/bootstrap_host_exec gnocchi_api su gnocchi -s /bin/bash -c /usr/bin/gnocchi-upgrade --sacks-number=128\", \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro\", \"/var/log/containers/gnocchi:/var/log/gnocchi\"], \"net\": \"host\", \"detach\": false, \"privileged\": false}', '--env=TRIPLEO_CONFIG_HASH=f264b1307217a5596bbcba581fee1833', '--net=host', '--privileged=false', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro', '--volume=/var/log/containers/gnocchi:/var/log/gnocchi', '192.168.24.1:8787/tripleoupstream/centos-binary-gnocchi-api:latest', '/usr/bin/bootstrap_host_exec', 'gnocchi_api', 'su', 'gnocchi', '-s', '/bin/bash', '-c', '/usr/bin/gnocchi-upgrade', '--sacks-number=128']. [1]",
            "Usage:",
            " su [options] [-] [USER [arg]...]",
            "Change the effective user id and group id to that of USER.",
            "A mere - implies -l. If USER not given, assume root.",
            "Options:",
            " -m, -p, --preserve-environment do not reset environment variables",
            " -g, --group <group> specify the primary group",
            " -G, --supp-group <group> specify a supplemental group",
            " -, -l, --login make the shell a login shell",
            " -c, --command <command> ...

[wes hayutin (weshayutin)]

 "stderr: /usr/lib/python2.7/site-packages/pymysql/cursors.py:166: Warning: (1831, u'Duplicate index `block_device_mapping_instance_uuid_virtual_name_device_name_idx`. This is deprecated and will be disallowed in a future release.')",

[Pradeep Kilambi (pkilambi)]

https://review.openstack.org/#/c/491826/ should fix this issue

[Giulio Fidente (gfidente)]

ack the above should fix (I proposed a similar change in https://review.openstack.org/#/c/492315/ before noticing it) but the job seems to be failing anyway on that same command with a different error

I am pushing a revert in https://review.openstack.org/#/c/492315/ just so we have both available in case the current fix shouldn't be sufficient

[Pradeep Kilambi (pkilambi)]

We needed few additional changes. But the scenario001 jobs are now passing:

https://review.openstack.org/#/c/491826/

lets get this in.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Pradeep Kilambi (<email address hidden>) on branch: master
Review: https://review.openstack.org/492315
Reason: Fix is merged, we can abandon this revert