Featureset022 fails to deploy on pike during .AllNodesDeploySteps.ControllerDeployment_Step1

[Gabriele Cerami]

logs at

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022-pike/9ecbf75/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2018-01-17_07_22_00

show that deploy fails after cliff subcommand returns error.

[Emilien Macchi (emilienm)]

how can we debug this one if we don't have overcloud logs?

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022-pike/9ecbf75/

[Sagi (Sergey) Shnaidman (sshnaidm)]

Overcloud hosts are not accessible by SSH, we can't fetch logs from there: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022-pike/9ecbf75/quickstart_collect_logs.txt.gz

[Michele Baldessari (michele)]

From https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022-pike/9ecbf75/undercloud/home/jenkins/failed_deployments.log.txt.gz we get:
[localhost]: FAILED! => {\n \"(outputs.stderr|default('')).split('\\n')|union(outputs.stdout_lines|default([]))\": [\n \"stdout: afa20bff30e1663b155ee691722de43051c4cf0d37fc697601ef7cf577487535\", \n \"\", \n \"stderr: \", \n \"stdout: 4de8c7231a0f500fb3ef74b5f539ba2b3db160115d13264df65a241137566622\", \n \"stdout: \", \n \"sequence item 33: expected string, list found\", \n \"Traceback (most recent call last):\", \n \" File \\\"/usr/lib/python2.7/site-packages/cliff/app.py\\\", line 400, in run_subcommand\", \n \" result = cmd.run(parsed_args)\", \n \" File \\\"/usr/lib/python2.7/site-packages/cliff/command.py\\\", line 137, in run\", \n \" return_code = self.take_action(parsed_args) or 0\", \n \" File \\\"/usr/lib/python2.7/site-packages/paunch/cmd.py\\\", line 75, in take_action\", \n \" labels=labels\", \n \" File \\\"/usr/lib/python2.7/site-packages/paunch/__init__.py\\\", line 54, in apply\", \n \" return builder.apply()\", \n \" File \\\"/usr/lib/python2.7/site-packages/paunch/builder/compose1.py\\\", line 66, in apply\", \n \" (cmd_stdout, cmd_stderr, returncode) = self.runner.execute(cmd)\", \n \" File \\\"/usr/lib/python2.7/site-packages/paunch/runner.py\\\", line 33, in execute\", \n \" LOG.debug('$ %s' % ' '.join(cmd))\", \n \"TypeError: sequence item 33: expected string, list found\", \n \" File \\\"/usr/bin/paunch\\\", line 10, in <module>\", \n \" sys.exit(main())\", \n \" File \\\"/usr/lib/python2.7/site-packages/paunch/main.py\\\", line 39, in main\", \n \" return myapp.run(argv)\", \n \" File \\\"/usr/lib/python2.7/site-packages/cliff/app.py\\\", line 279, in run\", \n \" result = self.run_subcommand(remainder)\"\n ], \n \"failed_when_result\": true\n}\n\tto retry, use: --limit @/var/lib/heat-config/heat-config-ansible/caf3f47b-95fa-43e7-89da-89b80047ee1e_playbook.retry\

[Bogdan Dobrelya (bogdando)]

A backport of https://review.openstack.org/#/c/522405/2/common/deploy-steps-tasks.yaml is missing

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/535326

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/535326
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=56448dafeda8db1916b2a65fd19bd70c7e6ff176
Submitter: Zuul
Branch: stable/pike

commit 56448dafeda8db1916b2a65fd19bd70c7e6ff176
Author: Ben Nemec <email address hidden>
Date: Wed Nov 22 17:09:13 2017 -0600

    Swap the order of stdout and stderr in debug output

    Generally this data is looked at because something failed, and in
    that case the relevant error is likely to be at the end of stderr.
    By concatenating the output stderr first and then stdout as we were
    it is possible for the stderr to get lost entirely in the failures
    list, and even if that doesn't happen it's best to output the
    relevant error right at the end of the output where people will
    see it. Previously it would be buried in the middle of the debug
    output.

    Closes-bug: #1743754

    Change-Id: I952fd1af5778ade1eb6b0599d983f98cadeb7f6f
    (cherry picked from commit 5595e7fc14430ee0636060afc82f909d4b3d04be)

[Bogdan Dobrelya (bogdando)]

IIUC, it should be fixed now?

[Bogdan Dobrelya (bogdando)]

As I can see, Featureset022 is still failing for the merged backport, so that was prolly not a (full) fix.

[Emilien Macchi (emilienm)]

Visible: here : http://logs.openstack.org/91/535291/2/check-tripleo/tripleo-ci-centos-7-ovb-containers-oooq/83e97f2/logs/overcloud-controller-0/var/log/journal.txt.gz#_Jan_22_16_55_14

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/536547

[Steve Baker (steve-stevebaker)]

The backport of Icf4a64ed76635e39bbb34c3a088c55e1f14fddca had an indentation error which caused the command to be ['/docker_puppet_apply.sh'] instead of '/docker_puppet_apply.sh'

This caused an error when paunch logged the command it would run. Making the paunch logging more robust would have made this easier to debug, as would capturing /var/log/tripleo-config in our CI jobs.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/536547
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ebf01289ce7b311ae6b18eb7c3bb11d46f7cc4b4
Submitter: Zuul
Branch: stable/pike

commit ebf01289ce7b311ae6b18eb7c3bb11d46f7cc4b4
Author: Steve Baker <email address hidden>
Date: Tue Jan 23 08:58:40 2018 +1300

    Correct indentation of docker_puppet_apply.sh run

    The backport of Icf4a64ed76635e39bbb34c3a088c55e1f14fddca had an
    indentation error which caused the command to be
    ['/docker_puppet_apply.sh'] instead of '/docker_puppet_apply.sh'

    This caused an error when paunch logged the command it would run.

    Making the paunch logging more robust would have made this easier to
    debug, as would capturing /var/log/tripleo-config in our CI jobs.

    Change-Id: I80734ccc842d5be01555ee5a863d0de9a8101d33
    Closes-Bug: #1743754

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.8 release.