Featureset022 fails to deploy on pike during .AllNodesDeploySteps.ControllerDeployment_Step1

Bug #1743754 reported by Gabriele Cerami on 2018-01-17
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Undecided
Unassigned
Pike
Critical
Steve Baker
wes hayutin (weshayutin) on 2018-01-17
Changed in tripleo:
assignee: Gabriele Cerami (gcerami) → nobody
Michele Baldessari (michele) wrote :

From https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022-pike/9ecbf75/undercloud/home/jenkins/failed_deployments.log.txt.gz we get:
[localhost]: FAILED! => {\n \"(outputs.stderr|default('')).split('\\n')|union(outputs.stdout_lines|default([]))\": [\n \"stdout: afa20bff30e1663b155ee691722de43051c4cf0d37fc697601ef7cf577487535\", \n \"\", \n \"stderr: \", \n \"stdout: 4de8c7231a0f500fb3ef74b5f539ba2b3db160115d13264df65a241137566622\", \n \"stdout: \", \n \"sequence item 33: expected string, list found\", \n \"Traceback (most recent call last):\", \n \" File \\\"/usr/lib/python2.7/site-packages/cliff/app.py\\\", line 400, in run_subcommand\", \n \" result = cmd.run(parsed_args)\", \n \" File \\\"/usr/lib/python2.7/site-packages/cliff/command.py\\\", line 137, in run\", \n \" return_code = self.take_action(parsed_args) or 0\", \n \" File \\\"/usr/lib/python2.7/site-packages/paunch/cmd.py\\\", line 75, in take_action\", \n \" labels=labels\", \n \" File \\\"/usr/lib/python2.7/site-packages/paunch/__init__.py\\\", line 54, in apply\", \n \" return builder.apply()\", \n \" File \\\"/usr/lib/python2.7/site-packages/paunch/builder/compose1.py\\\", line 66, in apply\", \n \" (cmd_stdout, cmd_stderr, returncode) = self.runner.execute(cmd)\", \n \" File \\\"/usr/lib/python2.7/site-packages/paunch/runner.py\\\", line 33, in execute\", \n \" LOG.debug('$ %s' % ' '.join(cmd))\", \n \"TypeError: sequence item 33: expected string, list found\", \n \" File \\\"/usr/bin/paunch\\\", line 10, in <module>\", \n \" sys.exit(main())\", \n \" File \\\"/usr/lib/python2.7/site-packages/paunch/main.py\\\", line 39, in main\", \n \" return myapp.run(argv)\", \n \" File \\\"/usr/lib/python2.7/site-packages/cliff/app.py\\\", line 279, in run\", \n \" result = self.run_subcommand(remainder)\"\n ], \n \"failed_when_result\": true\n}\n\tto retry, use: --limit @/var/lib/heat-config/heat-config-ansible/caf3f47b-95fa-43e7-89da-89b80047ee1e_playbook.retry\

Changed in tripleo:
assignee: nobody → Bogdan Dobrelya (bogdando)
status: Triaged → In Progress

Reviewed: https://review.openstack.org/535326
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=56448dafeda8db1916b2a65fd19bd70c7e6ff176
Submitter: Zuul
Branch: stable/pike

commit 56448dafeda8db1916b2a65fd19bd70c7e6ff176
Author: Ben Nemec <email address hidden>
Date: Wed Nov 22 17:09:13 2017 -0600

    Swap the order of stdout and stderr in debug output

    Generally this data is looked at because something failed, and in
    that case the relevant error is likely to be at the end of stderr.
    By concatenating the output stderr first and then stdout as we were
    it is possible for the stderr to get lost entirely in the failures
    list, and even if that doesn't happen it's best to output the
    relevant error right at the end of the output where people will
    see it. Previously it would be buried in the middle of the debug
    output.

    Closes-bug: #1743754

    Change-Id: I952fd1af5778ade1eb6b0599d983f98cadeb7f6f
    (cherry picked from commit 5595e7fc14430ee0636060afc82f909d4b3d04be)

tags: added: in-stable-pike
Bogdan Dobrelya (bogdando) wrote :

IIUC, it should be fixed now?

Changed in tripleo:
status: In Progress → Fix Committed
Bogdan Dobrelya (bogdando) wrote :

As I can see, Featureset022 is still failing for the merged backport, so that was prolly not a (full) fix.

Changed in tripleo:
assignee: Bogdan Dobrelya (bogdando) → nobody
status: Fix Committed → Triaged
Changed in tripleo:
status: Triaged → Invalid
importance: Critical → Undecided
milestone: queens-3 → none
Steve Baker (steve-stevebaker) wrote :

The backport of Icf4a64ed76635e39bbb34c3a088c55e1f14fddca had an indentation error which caused the command to be ['/docker_puppet_apply.sh'] instead of '/docker_puppet_apply.sh'

This caused an error when paunch logged the command it would run. Making the paunch logging more robust would have made this easier to debug, as would capturing /var/log/tripleo-config in our CI jobs.

Changed in tripleo:
milestone: none → queens-rc1

Reviewed: https://review.openstack.org/536547
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ebf01289ce7b311ae6b18eb7c3bb11d46f7cc4b4
Submitter: Zuul
Branch: stable/pike

commit ebf01289ce7b311ae6b18eb7c3bb11d46f7cc4b4
Author: Steve Baker <email address hidden>
Date: Tue Jan 23 08:58:40 2018 +1300

    Correct indentation of docker_puppet_apply.sh run

    The backport of Icf4a64ed76635e39bbb34c3a088c55e1f14fddca had an
    indentation error which caused the command to be
    ['/docker_puppet_apply.sh'] instead of '/docker_puppet_apply.sh'

    This caused an error when paunch logged the command it would run.

    Making the paunch logging more robust would have made this easier to
    debug, as would capturing /var/log/tripleo-config in our CI jobs.

    Change-Id: I80734ccc842d5be01555ee5a863d0de9a8101d33
    Closes-Bug: #1743754

This issue was fixed in the openstack/tripleo-heat-templates 7.0.8 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers