kolla_toolbox module sometimes ignores errors and changes

Bug #1844114 reported by Mark Goddard
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Fix Released
Medium
Mark Goddard
Queens
New
Medium
Unassigned
Rocky
New
Medium
Unassigned
Stein
Fix Committed
Medium
Mark Goddard
Train
Fix Released
Medium
Mark Goddard

Bug Description

The kolla_toolbox Ansible module executes as-hoc ansible commands in the kolla_toolbox container, and parses the output to make it look as if ansible-playbook executed the command. Currently however, this module fails to catch failures of the underlying command, and also shows tasks as 'ok' when the underlying command was changed. This has been tested both before and after the upgrade to ansible 2.8.

I used the following playbook to test this:

---
- hosts: localhost
  gather_facts: false
  tasks:
    - name: Test successful & unchanged
      kolla_toolbox:
        module_name: debug
        module_args:
          msg: hi
      register: result

    - name: Assert result is successful
      assert:
        that: result is successful

    - name: Assert result is not changed
      assert:
        that: result is not changed

    - name: Test successful & changed
      kolla_toolbox:
        module_name: command
        module_args:
          echo hi
      register: result

    - name: Assert result is successful
      assert:
        that: result is successful

    - name: Assert result is changed
      assert:
        that: result is changed
      # NOTE: Currently fails.
      ignore_errors: true

    - name: Test unsuccessful
      kolla_toolbox:
        module_name: command
        module_args:
          foo
      register: result
      ignore_errors: true

    - name: Assert result is failed
      assert:
        that: result is failed
      # NOTE: Currently fails.
      ignore_errors: true

    - name: Test invalid module parameters
      kolla_toolbox:
        module_name: debug
        module_args:
          foo: bar
      register: result
      ignore_errors: true

    - name: Assert result is failed
      assert:
        that: result is failed
      # NOTE: Currently fails.
      ignore_errors: true

Mark Goddard (mgoddard)
Changed in kolla-ansible:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/682340

Changed in kolla-ansible:
assignee: nobody → Mark Goddard (mgoddard)
status: Triaged → In Progress
Revision history for this message
Mark Goddard (mgoddard) wrote : Re: kolla_toolbox module ignores errors and changes

I think the situation is not quite as bad as I first thought. Modules sometimes produce JSON output, and we can catch this and parse the 'failed' and 'changed' fields. The problems I'm aware of are:

* modules like 'command' do not output in JSON format, so we can't detect whether they changed. We could detect their status, but currently do not.
* if we have an error like passing invalid module arguments, we do not get JSON format, and do not detect this error.

summary: - kolla_toolbox module ignores errors and changes
+ kolla_toolbox module sometimes ignores errors and changes
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/682344

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/682340
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=70b515bf1225e56b7df81677043d75be4bbb1ab4
Submitter: Zuul
Branch: master

commit 70b515bf1225e56b7df81677043d75be4bbb1ab4
Author: Mark Goddard <email address hidden>
Date: Mon Sep 16 11:27:48 2019 +0100

    Catch errors and changes in kolla_toolbox module

    The kolla_toolbox Ansible module executes as-hoc ansible commands in the
    kolla_toolbox container, and parses the output to make it look as if
    ansible-playbook executed the command. Currently however, this module
    sometimes fails to catch failures of the underlying command, and also
    sometimes shows tasks as 'ok' when the underlying command was changed.
    This has been tested both before and after the upgrade to ansible 2.8.

    This change fixes this issue by configuring ansible to emit output in
    JSON format, to make parsing simpler. We can now pick up errors and
    changes, and signal them to the caller.

    This change also adds an ansible playbook, tests/test-kolla-toolbox.yml,
    that can be executed to test the module. It's not currently integrated
    with any CI jobs.

    Note that this change cannot be backported as the JSON output callback
    plugin was added in Ansible 2.5.

    Change-Id: I8236dd4165f760c819ca972b75cbebc62015fada
    Closes-Bug: #1844114

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/stein)

Reviewed: https://review.opendev.org/682344
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=28273411a8fd76ff26b0507854efd0017dbb3520
Submitter: Zuul
Branch: stable/stein

commit 28273411a8fd76ff26b0507854efd0017dbb3520
Author: Mark Goddard <email address hidden>
Date: Mon Sep 16 12:30:23 2019 +0100

    Fix kolla_toolbox error handling

    Backport only.

    The kolla_toolbox Ansible module executes as-hoc ansible commands in the
    kolla_toolbox container, and parses the output to make it look as if
    ansible-playbook executed the command. Currently however, this module
    sometimes fails to catch failures of the underlying command, and also
    sometimes shows tasks as 'ok' when the underlying command was changed.

    This change introduces a check on the status field for non-JSON output,
    and assumes the module has changes on success.

    NOTE: The original patch on master
    (https://review.opendev.org/#/c/682340/) cannot be backported as it
    relies on Ansible 2.5+ in the kolla_toolbox container.

    Change-Id: Ib1e33f8e5c60d98b76a4eba9f405af89431d60bc
    Closes-Bug: 1844114

Revision history for this message
JK (jkk-2019) wrote :

TASK [mariadb : Creating haproxy mysql user] ***************************************************************************************************************************************************
fatal: [10.5.5.108]: FAILED! => {"changed": false, "msg": "Can not parse the inner module output: localhost | SUCCESS => {\n \"changed\": false, \n \"user\": \"haproxy\"\n}\n"}

Revision history for this message
Mark Goddard (mgoddard) wrote :

JK: As mentioned elsewhere, I think you are using the master branch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 9.0.0.0rc1

This issue was fixed in the openstack/kolla-ansible 9.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 8.1.0

This issue was fixed in the openstack/kolla-ansible 8.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.