get-occ-config.sh counts "wrong" cause of missing tonumber messing up the deployment

Bug #1768158 reported by James Slagle
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
James Slagle

Bug Description

When using the get-occ-config.sh on a deployment with more than 10 nodes, we noticed that it mixed up the metadata urls. So we had a few nodes configured correctly, and some had the wrong IPs deployed in the configs (split-stack), for example my_ip was wrong.

How reproducible:
deploy a split stack environment with more than 10 nodes.

Actual results:
Deployment sometimes got stuck because one node was missing (didn't get its
config because of wrong meta data) or deployment went fine but instead of 11 different nodes you see one node twice.

Expected results:
11 different nodes deployed and configured correctly

Additional info:
I checked how and why this would happen and i think i found the issue. In line
67 there is a sort_by on resource_name. If you check what it generates, you'll
notice that the sorting is *not* done as expected. This is cause it does string
sorting instead of integer sorting. The easy fix was to add a "tonumber" to
the sort_by: sort_by(.resource_name | tonumber)
That way node 11 (10 is the number of the node in the resource list) will end
up at the very end of the list, otherwise it showed up as 3rd node (0, 1, 10, 2).

Demo:
(undercloud) [stack@director ~]$ openstack stack resource list 55a9e155-edab-4734-a655-523339d0c8ac -c resource_name -c physical_resource_id
+---------------+--------------------------------------+
| resource_name | physical_resource_id |
+---------------+--------------------------------------+
| 10 | e4ee76de-9794-4bf8-81d5-3fd5c54fc33e |
| 1 | 46f3d8f0-8d5d-47a7-ad59-a2487ce55fb5 |
| 0 | ddf8db32-4bf3-4635-9020-28b07c21d66a |
| 3 | 48745fa1-f5e2-41ca-8a3e-e9da4cf2bf4e |
| 2 | f4a586f8-e93a-49aa-bf88-97436da05cb6 |
| 5 | 137a14db-a641-4782-b5c0-5f554cf76f46 |
| 4 | afe87089-683e-4c8d-bf43-92a11c0b1f96 |
| 7 | bc84c066-827a-4bfe-b830-3532ffd24875 |
| 6 | f80f41b9-f89b-443e-b0bf-ee99bea43654 |
| 9 | c0df0d59-9c44-4712-abd2-54f05cdd59c2 |
| 8 | da466290-adac-4c53-b930-d42d634c8436 |
+---------------+--------------------------------------+
(undercloud) [stack@director ~]$ openstack stack resource list 55a9e155-edab-4734-a655-523339d0c8ac -c resource_name -c physical_resource_id -f json | jq -r "sort_by(.resource_name) | .[] | .physical_resource_id"
ddf8db32-4bf3-4635-9020-28b07c21d66a
46f3d8f0-8d5d-47a7-ad59-a2487ce55fb5
e4ee76de-9794-4bf8-81d5-3fd5c54fc33e
f4a586f8-e93a-49aa-bf88-97436da05cb6
48745fa1-f5e2-41ca-8a3e-e9da4cf2bf4e
afe87089-683e-4c8d-bf43-92a11c0b1f96
137a14db-a641-4782-b5c0-5f554cf76f46
f80f41b9-f89b-443e-b0bf-ee99bea43654
bc84c066-827a-4bfe-b830-3532ffd24875
da466290-adac-4c53-b930-d42d634c8436
c0df0d59-9c44-4712-abd2-54f05cdd59c2
(undercloud) [stack@director ~]$ openstack stack resource list 55a9e155-edab-4734-a655-523339d0c8ac -c resource_name -c physical_resource_id -f json | jq -r "sort_by(.resource_name | tonumber) | .[] | .physical_resource_id"
ddf8db32-4bf3-4635-9020-28b07c21d66a
46f3d8f0-8d5d-47a7-ad59-a2487ce55fb5
f4a586f8-e93a-49aa-bf88-97436da05cb6
48745fa1-f5e2-41ca-8a3e-e9da4cf2bf4e
afe87089-683e-4c8d-bf43-92a11c0b1f96
137a14db-a641-4782-b5c0-5f554cf76f46
f80f41b9-f89b-443e-b0bf-ee99bea43654
bc84c066-827a-4bfe-b830-3532ffd24875
da466290-adac-4c53-b930-d42d634c8436
c0df0d59-9c44-4712-abd2-54f05cdd59c2
e4ee76de-9794-4bf8-81d5-3fd5c54fc33e

So you can clearly see the difference with and without "tonumber".

Changed in tripleo:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → James Slagle (james-slagle)
milestone: none → rocky-2
tags: added: pike-backport-potential queens-backport-potential
tags: added: ocata-backport-potential
tags: added: newton-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/565369

Changed in tripleo:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/565369
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=49d072133563dde3c7693e8dacbfdaafac121329
Submitter: Zuul
Branch: master

commit 49d072133563dde3c7693e8dacbfdaafac121329
Author: James Slagle <email address hidden>
Date: Mon Apr 30 20:03:19 2018 -0400

    Convert resource_name to number

    When sorting by resource_name, first convert it to a number so that
    it's sorted correctly. Otherwise, deployments with > 10 nodes could
    configure nodes out of order.

    Change-Id: I604428dacd63140f4e89b45c55f7eb859df27fe7
    Closes-Bug: #1768158

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/570418

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/570419

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/570420

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.openstack.org/570418
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=e508323f6b73e2d02de53229a3861fb33960134a
Submitter: Zuul
Branch: stable/queens

commit e508323f6b73e2d02de53229a3861fb33960134a
Author: James Slagle <email address hidden>
Date: Mon Apr 30 20:03:19 2018 -0400

    Convert resource_name to number

    When sorting by resource_name, first convert it to a number so that
    it's sorted correctly. Otherwise, deployments with > 10 nodes could
    configure nodes out of order.

    Change-Id: I604428dacd63140f4e89b45c55f7eb859df27fe7
    Closes-Bug: #1768158
    (cherry picked from commit 49d072133563dde3c7693e8dacbfdaafac121329)

tags: added: in-stable-queens
tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/pike)

Reviewed: https://review.openstack.org/570419
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=568c09eddf6e86586267be7a4741505e45c6cb26
Submitter: Zuul
Branch: stable/pike

commit 568c09eddf6e86586267be7a4741505e45c6cb26
Author: James Slagle <email address hidden>
Date: Mon Apr 30 20:03:19 2018 -0400

    Convert resource_name to number

    When sorting by resource_name, first convert it to a number so that
    it's sorted correctly. Otherwise, deployments with > 10 nodes could
    configure nodes out of order.

    Change-Id: I604428dacd63140f4e89b45c55f7eb859df27fe7
    Closes-Bug: #1768158
    (cherry picked from commit 49d072133563dde3c7693e8dacbfdaafac121329)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/ocata)

Reviewed: https://review.openstack.org/570420
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=bffc619706c6bd751bdfb279a9608759b82e6253
Submitter: Zuul
Branch: stable/ocata

commit bffc619706c6bd751bdfb279a9608759b82e6253
Author: James Slagle <email address hidden>
Date: Mon Apr 30 20:03:19 2018 -0400

    Convert resource_name to number

    When sorting by resource_name, first convert it to a number so that
    it's sorted correctly. Otherwise, deployments with > 10 nodes could
    configure nodes out of order.

    Change-Id: I604428dacd63140f4e89b45c55f7eb859df27fe7
    Closes-Bug: #1768158
    (cherry picked from commit 49d072133563dde3c7693e8dacbfdaafac121329)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 8.0.3

This issue was fixed in the openstack/tripleo-heat-templates 8.0.3 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 9.0.0.0b3

This issue was fixed in the openstack/tripleo-heat-templates 9.0.0.0b3 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 6.2.15

This issue was fixed in the openstack/tripleo-heat-templates 6.2.15 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 7.0.14

This issue was fixed in the openstack/tripleo-heat-templates 7.0.14 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.