tripleo

standalone upgrade do not raise if upgrade_tasks failed

Bug #1869230 reported by Emilien Macchi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Emilien Macchi
tripleo ussuri-3 "tripleo ussuri-3"

Bug Description

As you can observe, the upgrade process moves on even if the upgrade_tasks failed to pull latest haproxy image:

TASK [Pull latest haproxy images] **********************************************
Thursday 26 March 2020 16:03:43 +0000 (0:00:00.170) 0:02:46.313 ********
fatal: [undercloud]: FAILED! => changed=true
  cmd:
  - podman
  - pull
  - 192.168.121.233:8787/tripleomaster/centos-binary-haproxy:current-tripleo
  delta: '0:00:00.075762'
  end: '2020-03-26 16:03:43.226010'
  msg: non-zero return code
  rc: 125
  start: '2020-03-26 16:03:43.150248'
  stderr: |-
    Trying to pull 192.168.121.233:8787/tripleomaster/centos-binary-haproxy:current-tripleo...
      error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL /v
2/tripleomaster/centos-binary-haproxy/manifests/current-tripleo was not found on this server.</p>\n</body></html>\n"
    Error: error pulling image "192.168.121.233:8787/tripleomaster/centos-binary-haproxy:current-tripleo": unable to pull 192.168.121.233:8787/tripleomaster/centos-binary-haproxy:current-tripleo: unable to pull image: Error initializing source docker://19
2.168.121.233:8787/tripleomaster/centos-binary-haproxy:current-tripleo: Error reading manifest current-tripleo in 192.168.121.233:8787/tripleomaster/centos-binary-haproxy: error parsing HTTP 404 response body: invalid character '<' looking for beginning o
f value: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL /v2/tripleomaster/centos-binary-haproxy/manifests/current-tripleo was not found on this ser
ver.</p>\n</body></html>\n"
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
undercloud : ok=340 changed=130 unreachable=0 failed=1 skipped=149 rescued=0 ignored=0
Thursday 26 March 2020 16:03:43 +0000 (0:00:00.266) 0:02:46.580 ********
===============================================================================
tripleo_container_rm : stop and disable cinder_volume ------------------ 11.16s
Ensure TripleO prerequisite packages are installed --------------------- 10.89s
Render all_nodes data as group_vars for overcloud ----------------------- 6.04s
tripleo_container_rm : stop and disable rabbitmq ------------------------ 2.82s
tripleo_container_rm : stop and disable mysql --------------------------- 2.06s
Install libibverbs (https://bugs.launchpad.net/tripleo/+bug/1817743) ---- 1.21s
tripleo_container_tag : Pull 192.168.121.233:8787/tripleomaster/centos-binary-cinder-volume:current-tripleo image --- 1.14s
Remove openstack-nova-compute and python-nova package during upgrade ---- 1.10s
tripleo_podman : ensure podman and deps are installed ------------------- 1.08s
Pull latest cinder_volume images ---------------------------------------- 1.08s
Process rhosp-openvswitch layered package for new version number -------- 1.05s
check if an upgrade is required ----------------------------------------- 1.04s
tripleo_upgrade_hiera : write the updated tripleo-upgrade hiera data ---- 0.98s
tripleo_container_rm : stop and disable ovn_north_db_server ------------- 0.97s
tripleo_upgrade_hiera : write the updated tripleo-upgrade hiera data ---- 0.90s
tripleo_container_rm : stop and disable ovn_northd ---------------------- 0.87s
tripleo_container_rm : stop and disable ovn_south_db_server ------------- 0.87s
tripleo_container_rm : stop and disable cinder_volume healthcheck ------- 0.78s
tripleo_upgrade_hiera : create the hiera file when no file or empty file. --- 0.76s
ensure we get the right selinux context --------------------------------- 0.67s
2020-03-26 16:03:43.552 73495 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleoef2uxua9 ] cleaned up
2020-03-26 16:03:43.552 73495 ERROR tripleoclient.utils.utils [-] Ansible execution failed. playbook: upgrade_steps_playbook.yaml, Run Status: failed, Return Code: 2
2020-03-26 16:03:43.552 73495 INFO tripleoclient.utils.utils [-] Ansible execution success. playbook: upgrade_steps_playbook.yaml
2020-03-26 16:03:43.553 73495 INFO tripleoclient.utils.utils [-] Running Ansible playbook: deploy_steps_playbook.yaml, Working directory: /home/vagrant/standalone-ansible-enatcpn3, Playbook directory: /home/vagrant/standalone-ansible-enatcpn3

PLAY [Gather facts from undercloud] ********************************************

PLAY [Gather facts from overcloud] *********************************************

PLAY [Load global variables] ***************************************************

TASK [include_vars] ************************************************************
Thursday 26 March 2020 16:03:44 +0000 (0:00:00.118) 0:00:00.118 ********

Emilien Macchi (emilienm)
Changed in tripleo:
status: New → Triaged
importance: Undecided → Critical
milestone: none → ussuri-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (master)

Fix proposed to branch: master
Review: https://review.opendev.org/715273

Changed in tripleo:
assignee: nobody → Emilien Macchi (emilienm)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (master)

Reviewed: https://review.opendev.org/715273
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=9c655a97b7fd1abffe0fcec69c66c581b2ac5228
Submitter: Zuul
Branch: master

commit 9c655a97b7fd1abffe0fcec69c66c581b2ac5228
Author: Emilien Macchi <email address hidden>
Date: Thu Mar 26 13:14:27 2020 -0400

    tripleo_deploy: finish the loop of operations early if wrong rc

    If one of the operations running Ansible playbooks returns a wrong rc,
    leave the loop and go to final tasks before raising.

    It'll prevent the upgrade tasks to fail but the other playbooks (e.g.
    deploy) to keep running.

    Closes-Bug: #1869230
    Change-Id: I5b1bb964ed290c860e66074202c45d7dd09bf29c

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/python-tripleoclient 13.2.0

This issue was fixed in the openstack/python-tripleoclient 13.2.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.