Ocata -> Pike overcloud upgrade: major-upgrade-composable-steps-docker fails with An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: cannot import name shlex_quote

Bug #1729546 reported by Marius Cornea
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Won't Fix
High
Unassigned

Bug Description

Ocata -> Pike upgrade: major-upgrade-composable-steps-docker fails with An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: cannot import name shlex_quote:

This is the failed upgrade tasks in the failures list output

overcloud.AllNodesDeploySteps.ControllerUpgrade_Step3.1:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: 77bef0af-a037-46a5-9f8a-b8d69dc07d5c
  status: CREATE_FAILED
  status_reason: |
    Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |

    PLAY [localhost] ***************************************************************

    TASK [Gathering Facts] *********************************************************
    ok: [localhost]

    TASK [Check if httpd is deployed] **********************************************
    fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["systemctl", "is-enabled", "httpd"], "delta": "0:00:00.008017", "end": "2017-10-19 20:22:50.996250", "failed": true, "rc": 1, "start": "2017-10-19 20:22:50.988233", "stderr": "", "stderr_lines": [], "stdout": "disabled", "stdout_lines": ["disabled"]}
    ...ignoring

    TASK [Ensure mod_ssl package is installed] *************************************
    changed: [localhost]

    TASK [Check if cinder_api is deployed] *****************************************
    fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["systemctl", "is-enabled", "openstack-cinder-api"], "delta": "0:00:00.008898", "end": "2017-10-19 20:22:58.423753", "failed": true, "rc": 1, "start": "2017-10-19 20:22:58.414855", "stderr": "", "stderr_lines": [], "stdout": "disabled", "stdout_lines": ["disabled"]}
    ...ignoring

    TASK [Check if cinder_scheduler is deployed] ***********************************
    changed: [localhost]

    TASK [Check is heat_api is deployed] *******************************************
    fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["systemctl", "is-enabled", "openstack-heat-api"], "delta": "0:00:00.005772", "end": "2017-10-19 20:22:58.858121", "failed": true, "rc": 1, "start": "2017-10-19 20:22:58.852349", "stderr": "", "stderr_lines": [], "stdout": "disabled", "stdout_lines": ["disabled"]}
    ...ignoring

    TASK [Check if heat_api_cfn is deployed] ***************************************
    fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["systemctl", "is-enabled", "openstack-heat-api-cfn"], "delta": "0:00:00.005196", "end": "2017-10-19 20:22:59.053675", "failed": true, "rc": 1, "start": "2017-10-19 20:22:59.048479", "stderr": "", "stderr_lines": [], "stdout": "disabled", "stdout_lines": ["disabled"]}
    ...ignoring

    TASK [Check if heat_api_cloudwatch is deployed] ********************************
    fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["systemctl", "is-enabled", "openstack-heat-api-cloudwatch"], "delta": "0:00:00.005477", "end": "2017-10-19 20:22:59.257828", "failed": true, "rc": 1, "start": "2017-10-19 20:22:59.252351", "stderr": "", "stderr_lines": [], "stdout": "disabled", "stdout_lines": ["disabled"]}
    ...ignoring

    TASK [Check if neutron_server is deployed] *************************************
    changed: [localhost]

    TASK [get bootstrap nodeid] ****************************************************
    changed: [localhost]

    TASK [set is_bootstrap_node fact] **********************************************
    ok: [localhost]

    TASK [Stop pacemaker cluster] **************************************************
    changed: [localhost]

    TASK [Check for mongodb service] ***********************************************
    ok: [localhost]

    TASK [Install docker packages on upgrade if missing] ***************************
    changed: [localhost]

    TASK [Upgrade os-net-config] ***************************************************
    changed: [localhost]

    TASK [take new os-net-config parameters into account now] **********************
    changed: [localhost]

    TASK [Update all packages] *****************************************************
    changed: [localhost]

    TASK [blank ipv6 rule before activating ipv6 firewall.] ************************
    An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: cannot import name shlex_quote
    fatal: [localhost]: FAILED! => {"failed": true, "msg": "Unexpected failure during module execution.", "stdout": ""}
     to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/0161d598-50c6-442f-9eeb-64a1ba779abf_playbook.retry

    PLAY RECAP *********************************************************************
    localhost : ok=17 changed=14 unreachable=0 failed=1

  deploy_stderr: |

Changed in tripleo:
status: New → Triaged
milestone: none → queens-2
tags: added: pike-backport-potential
Revision history for this message
Alex Schultz (alex-schultz) wrote :

I believe this was identified as an issue with the ansible version. What version was used with this report?

Revision history for this message
Marius Cornea (mcornea) wrote :

We're still investigating if the ansible 2.4.1 actually fixes the issue during the upgrade workflow. I filed this bug to keep track of this patch - https://review.openstack.org/#/c/517025/ which addresses the issue.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/517025
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=0c1ac1d752aaf88832b34e165f7d147e2304ff1c
Submitter: Zuul
Branch: master

commit 0c1ac1d752aaf88832b34e165f7d147e2304ff1c
Author: Marius Cornea <email address hidden>
Date: Wed Nov 1 17:08:31 2017 +0100

    Update ansible before the ansible upgrade tasks

    In case an ansible update is available during the upgrade then
    the ansible package gets updated to a new version by the ansible
    tasks. This could potentially lead to issues as the one described
    in LP#1729546. This change updates the ansible package via yum
    before starting the ansible upgrade tasks in order to avoid having
    ansible updating itself.

    Related-bug: 1729546

    Change-Id: I2ea0aa1f670053578996018663c9fa52dec14b77

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/519348

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/pike)

Reviewed: https://review.openstack.org/519348
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=af701f29f9b1bd0d19dc356d0071dce18f21cdf9
Submitter: Zuul
Branch: stable/pike

commit af701f29f9b1bd0d19dc356d0071dce18f21cdf9
Author: Marius Cornea <email address hidden>
Date: Wed Nov 1 17:08:31 2017 +0100

    Update ansible before the ansible upgrade tasks

    In case an ansible update is available during the upgrade then
    the ansible package gets updated to a new version by the ansible
    tasks. This could potentially lead to issues as the one described
    in LP#1729546. This change updates the ansible package via yum
    before starting the ansible upgrade tasks in order to avoid having
    ansible updating itself.

    Related-bug: 1729546

    Change-Id: I2ea0aa1f670053578996018663c9fa52dec14b77
    (cherry picked from commit 0c1ac1d752aaf88832b34e165f7d147e2304ff1c)

tags: added: in-stable-pike
Changed in tripleo:
milestone: queens-2 → queens-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart-extras (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/528261

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart-extras (master)

Reviewed: https://review.openstack.org/528261
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=90c2eb3929d699bd3c2e25556851aba42ce2828e
Submitter: Zuul
Branch: master

commit 90c2eb3929d699bd3c2e25556851aba42ce2828e
Author: marios <email address hidden>
Date: Fri Dec 15 15:00:18 2017 +0200

    Add ansible update into UpgradeInitCommand of repo template

    In I2ea0aa1f670053578996018663c9fa52dec14b77 we added an explicit
    update of ansible into the UpgradeInitCommand. Here just adding
    to the overcloud-repo template.

    Change-Id: Iaae7eb9f100122c28529ceebf3aefe3f488f7179
    Related-Bug: 1729546

Changed in tripleo:
milestone: queens-3 → queens-rc1
Changed in tripleo:
milestone: queens-rc1 → rocky-1
Changed in tripleo:
milestone: rocky-1 → rocky-2
Changed in tripleo:
milestone: rocky-2 → rocky-3
Changed in tripleo:
milestone: rocky-3 → rocky-rc1
Changed in tripleo:
milestone: rocky-rc1 → stein-1
Changed in tripleo:
milestone: stein-1 → stein-2
Changed in tripleo:
milestone: stein-2 → stein-3
Changed in tripleo:
milestone: stein-3 → stein-rc1
Changed in tripleo:
milestone: stein-rc1 → train-1
Revision history for this message
Natal Ngétal (hobbestigrou) wrote :

This ticket is always relevant? A patch was proposed and merged in master, so I suppose it's possible to close this ticket.

Changed in tripleo:
milestone: train-1 → train-2
Changed in tripleo:
milestone: train-2 → train-3
Changed in tripleo:
milestone: train-3 → ussuri-1
Revision history for this message
mathieu bultel (mat-bultel) wrote :

I think this bug is not relevant anymore.
I'm going to close it, if anybody wants to reopen it fell free

Changed in tripleo:
status: Triaged → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.