OpenShift master nodes scaleup fails with undefined ansible variable

Bug #1812962 reported by Martin André on 2019-01-23
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
High
Martin André

Bug Description

Trying to scale out OpenShift master nodes from TripleO fails with the following error:

TASK [openshift_service_catalog : template] ************************************
[0;31mfatal: [openshift-master-2]: FAILED! => {"msg": "The field 'vars' has an invalid value, which includes an undefined variable. The error was: 'openshift_master_etcd_urls' is undefined\n\nThe error appears to have been in '/usr/share/ansible/openshift-ansible/roles/openshift_service_catalog/tasks/install.yml': line 102, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n## api server\n- template:\n ^ here\n"}[0m

PLAY RECAP *********************************************************************
[0;32mlocalhost[0m : [0;32mok=36 [0m changed=0 unreachable=0 failed=0
[0;33mopenshift-infra-0[0m : [0;32mok=41 [0m [0;33mchanged=3 [0m unreachable=0 failed=0
[0;33mopenshift-infra-1[0m : [0;32mok=41 [0m [0;33mchanged=3 [0m unreachable=0 failed=0
[0;33mopenshift-infra-2[0m : [0;32mok=41 [0m [0;33mchanged=3 [0m unreachable=0 failed=0
[0;33mopenshift-master-1[0m : [0;32mok=72 [0m [0;33mchanged=8 [0m unreachable=0 failed=0
[0;31mopenshift-master-2[0m : [0;32mok=690 [0m [0;33mchanged=155 [0m unreachable=0 [0;31mfailed=1 [0m
[0;33mopenshift-master-3[0m : [0;32mok=569 [0m [0;33mchanged=184 [0m unreachable=0 failed=0
[0;33mopenshift-worker-0[0m : [0;32mok=41 [0m [0;33mchanged=3 [0m unreachable=0 failed=0
[0;33mopenshift-worker-1[0m : [0;32mok=41 [0m [0;33mchanged=3 [0m unreachable=0 failed=0
[0;33mopenshift-worker-2[0m : [0;32mok=41 [0m [0;33mchanged=3 [0m unreachable=0 failed=0

INSTALLER STATUS ***************************************************************
[0;32mLoad Balancer Install : Complete (0:00:00)[0m
[0;32mInitialization : Complete (0:03:00)[0m
[0;32mHealth Check : Complete (0:00:18)[0m
[0;32mNode Bootstrap Preparation : Complete (0:01:31)[0m
[0;32metcd Install : Complete (0:01:04)[0m
[0;32mMaster Install : Complete (0:04:20)[0m
[0;32mMaster Additional Install : Complete (0:00:57)[0m
[0;32mNode Join : Complete (0:00:15)[0m
[0;32mGlusterFS Install : Complete (0:02:15)[0m
[0;32mHosted Install : Complete (0:01:37)[0m
[0;32mCluster Monitoring Operator : Complete (0:00:16)[0m
[0;32mWeb Console Install : Complete (0:00:46)[0m
[0;32mConsole Install : Complete (0:00:16)[0m
[0;32mmetrics-server Install : Complete (0:00:01)[0m
[0;31mService Catalog Install : In Progress (0:00:38)[0m
 This phase can be restarted by running: playbooks/openshift-service-catalog/config.yml

Failure summary:

  1. Hosts: openshift-master-2
     Play: Service Catalog
     Task: openshift_service_catalog : template
     Message: [0;31mThe field 'vars' has an invalid value, which includes an undefined variable. The error was: 'openshift_master_etcd_urls' is undefined[0m
               [0;31m[0m
               [0;31mThe error appears to have been in '/usr/share/ansible/openshift-ansible/roles/openshift_service_catalog/tasks/install.yml': line 102, column 3, but may[0m
               [0;31mbe elsewhere in the file depending on the exact syntax problem.[0m
               [0;31m[0m
               [0;31mThe offending line appears to be:[0m
               [0;31m[0m
               [0;31m## api server[0m
               [0;31m- template:[0m
               [0;31m ^ here[0m
               [0;31m[0m

Originally reported at https://bugzilla.redhat.com/show_bug.cgi?id=1653348

Related fix proposed to branch: master
Review: https://review.openstack.org/632639

Changed in tripleo:
status: Triaged → In Progress

Reviewed: https://review.openstack.org/632638
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1febc8b7b307cb5be59ca69bd1890f4817b8421f
Submitter: Zuul
Branch: master

commit 1febc8b7b307cb5be59ca69bd1890f4817b8421f
Author: Martin André <email address hidden>
Date: Tue Jan 22 09:47:22 2019 +0100

    Store nodes information in a dict

    Re-using the same variable for each role was not very clean and forced
    us to re-check which master nodes were new in the deployment. Instead,
    use a dict for `role_nodes` and `new_role_nodes` with keys being the
    role name.

    This will allow us to re-generate updated inventory in later step
    without needing to set the facts once again.

    Related-Bug: #1812962
    Change-Id: I21ed56a1c4d43a3e0945c8b74dd94ccf89dc05b0

Reviewed: https://review.openstack.org/632639
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ff36d44af5a42945ac1b76a798af90a87347975c
Submitter: Zuul
Branch: master

commit ff36d44af5a42945ac1b76a798af90a87347975c
Author: Martin André <email address hidden>
Date: Tue Jan 22 10:05:30 2019 +0100

    Generate post-deployment openshift-ansible inventory

    This allows to run custom openshift-ansible playbooks with an inventory
    that reflects the deployment status.

    Related-Bug: #1812962
    Change-Id: I94b6ae04a3e31fe0b7951df2446cc91b5842cd77

Reviewed: https://review.openstack.org/632640
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=8a818ab22618321fa78a9414f458b3029ac75be6
Submitter: Zuul
Branch: master

commit 8a818ab22618321fa78a9414f458b3029ac75be6
Author: Martin André <email address hidden>
Date: Tue Jan 22 18:08:14 2019 +0100

    Apply changes to cluster using updated inventory

    It could cause issue when being on inventory containing nodes marked as
    new.

    Specifically, it caused an undefined ansible variable error when
    re-running the deploy playbook on an inventory that contained nodes in
    the new_masters group.

    Closes-Bug: #1812962
    Change-Id: Iaf9403b5f60f06769d94317d261de8cad94274ab

Changed in tripleo:
status: In Progress → Fix Released

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/632997

Reviewed: https://review.openstack.org/632996
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=9a07d00c1c51643ae8bfd262473ee48168f2b2f1
Submitter: Zuul
Branch: stable/rocky

commit 9a07d00c1c51643ae8bfd262473ee48168f2b2f1
Author: Martin André <email address hidden>
Date: Tue Jan 22 09:47:22 2019 +0100

    Store nodes information in a dict

    Re-using the same variable for each role was not very clean and forced
    us to re-check which master nodes were new in the deployment. Instead,
    use a dict for `role_nodes` and `new_role_nodes` with keys being the
    role name.

    This will allow us to re-generate updated inventory in later step
    without needing to set the facts once again.

    Related-Bug: #1812962
    Change-Id: I21ed56a1c4d43a3e0945c8b74dd94ccf89dc05b0
    (cherry picked from commit 1febc8b7b307cb5be59ca69bd1890f4817b8421f)

tags: added: in-stable-rocky

Reviewed: https://review.openstack.org/632997
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=3ce479fb17e659cd3779e1f25c4c9ff3e6de50c6
Submitter: Zuul
Branch: stable/rocky

commit 3ce479fb17e659cd3779e1f25c4c9ff3e6de50c6
Author: Martin André <email address hidden>
Date: Tue Jan 22 10:05:30 2019 +0100

    Generate post-deployment openshift-ansible inventory

    This allows to run custom openshift-ansible playbooks with an inventory
    that reflects the deployment status.

    Related-Bug: #1812962
    Change-Id: I94b6ae04a3e31fe0b7951df2446cc91b5842cd77
    (cherry picked from commit ff36d44af5a42945ac1b76a798af90a87347975c)

Reviewed: https://review.openstack.org/632998
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1dfc4267f7b95fbd9a394d499087a75cbbe75975
Submitter: Zuul
Branch: stable/rocky

commit 1dfc4267f7b95fbd9a394d499087a75cbbe75975
Author: Martin André <email address hidden>
Date: Tue Jan 22 18:08:14 2019 +0100

    Apply changes to cluster using updated inventory

    It could cause issue when being on inventory containing nodes marked as
    new.

    Specifically, it caused an undefined ansible variable error when
    re-running the deploy playbook on an inventory that contained nodes in
    the new_masters group.

    Closes-Bug: #1812962
    Change-Id: Iaf9403b5f60f06769d94317d261de8cad94274ab
    (cherry picked from commit 8a818ab22618321fa78a9414f458b3029ac75be6)

This issue was fixed in the openstack/tripleo-heat-templates 10.4.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers