tripleo

HA deployment with uppercase hostnames can fail

Bug #1773219 reported by Michele Baldessari
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Michele Baldessari
tripleo rocky-2

Bug Description

Via https://bugzilla.redhat.com/show_bug.cgi?id=1564654 if the hostname has uppercase letters we can hit issues:

Debug: try 15/20: /usr/sbin/pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20180405-8-1sqw3dc property set --node TEST-STACK34-controller-1 redis-role=true
Debug: Error: Error: unable to set attribute redis-role
Could not map name=TEST-STACK34-controller-1 to a UUID
while the name in the cluster is test-stack34-controller-1

the stack name has capital letters vs lower case how the node is registered in the cluster.

In bundles we use the hiera keys of short_node_names to set the pcs properties. This can contain upper case character which would then break when pushing them to the cib which has only lower case node names

OpenStack Infra (hudson-openstack)
Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
Michele Baldessari (michele) wrote :

https://review.openstack.org/570413

Michele Baldessari (michele)
tags: added: pike-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/570484

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (master)

Reviewed: https://review.openstack.org/570484
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=3345557d7b64b29ccee258de8bcd6cfbfa88291e
Submitter: Zuul
Branch: master

commit 3345557d7b64b29ccee258de8bcd6cfbfa88291e
Author: Michele Baldessari <email address hidden>
Date: Fri May 25 01:56:32 2018 +0200

    Lowercase any hostname comparisons in bootstrap_* scripts

    If you deploy a stack with mixed-case hostnames things will break
    because no task that is supposed to run on boostrap nodes will run
    due to the following code:
    HOSTNAME=$(/bin/hostname -s)
    SERVICE_NODEID=$(/bin/hiera -c /etc/puppet/hiera.yaml
    "${SERVICE_NAME}_short_bootstrap_node_name")
    if [[ "$HOSTNAME" == "$SERVICE_NODEID" ]]; then
    ...

    The hiera key might contain mixed-case letters whereas the hostname
    won't and the end result is going to be that no bootstrap tasks
    will run on any nodes and, amongst other things, no database tables
    will be created, making all services unusable.

    Since we use bash explicitely we can leverage the ${var,,} expression
    for this.

    Change-Id: Ie240b8a4217827dd8ade82479a828817d63143ba
    Related-Bug: #1773219

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/570844

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (stable/queens)

Reviewed: https://review.openstack.org/570844
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=57dea6a3c77456b96d84d3e4f3d2881424194765
Submitter: Zuul
Branch: stable/queens

commit 57dea6a3c77456b96d84d3e4f3d2881424194765
Author: Michele Baldessari <email address hidden>
Date: Fri May 25 01:56:32 2018 +0200

    Lowercase any hostname comparisons in bootstrap_* scripts

    If you deploy a stack with mixed-case hostnames things will break
    because no task that is supposed to run on boostrap nodes will run
    due to the following code:
    HOSTNAME=$(/bin/hostname -s)
    SERVICE_NODEID=$(/bin/hiera -c /etc/puppet/hiera.yaml
    "${SERVICE_NAME}_short_bootstrap_node_name")
    if [[ "$HOSTNAME" == "$SERVICE_NODEID" ]]; then
    ...

    The hiera key might contain mixed-case letters whereas the hostname
    won't and the end result is going to be that no bootstrap tasks
    will run on any nodes and, amongst other things, no database tables
    will be created, making all services unusable.

    Since we use bash explicitely we can leverage the ${var,,} expression
    for this.

    Change-Id: Ie240b8a4217827dd8ade82479a828817d63143ba
    Related-Bug: #1773219
    (cherry picked from commit 3345557d7b64b29ccee258de8bcd6cfbfa88291e)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (master)

Reviewed: https://review.openstack.org/570413
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=f2484a0bf9e64ff84979abe9b70e7f65957b9d26
Submitter: Zuul
Branch: master

commit f2484a0bf9e64ff84979abe9b70e7f65957b9d26
Author: Michele Baldessari <email address hidden>
Date: Thu May 24 18:47:02 2018 +0200

    Fix up property names in case of mixed case hostnames

    When deploying a stack that containes mixed-case hostnames
    the following error might be triggered:
    Debug: try 15/20: /usr/sbin/pcs -f
    /var/lib/pacemaker/cib/puppet-cib-backup20180405-8-1sqw3dc property set
    --node TEST-STACK34-controller-1 redis-role=true
    Debug: Error: Error: unable to set attribute redis-role
    Could not map name=TEST-STACK34-controller-1 to a UUID
    while the name in the cluster is test-stack34-controller-1

    This used to work pre-bundles because we used the facter provided
    $::hostname variable which was lower-cased for us. With bundles we
    switched to setting cluster properties from the service bootstrap nodes
    and so we used the '<service>_short_node_names' hiera key which might
    contain mixed-case hostnames.

    In order to fix this we just downcase() the short_node_names hiera
    string that we loop on so we can get the same behaviour we had on bare
    metal.

    Tested on an env with mixed-case hostnames:
    [root@uppercaseovercloud-controller-0 keystone]# hiera -c /etc/puppet/hiera.yaml rabbitmq_short_node_names
    ["UPPERCASEOverCloud-controller-0",
     "UPPERCASEOverCloud-controller-1",
     "UPPERCASEOverCloud-controller-2"]

    Cluster pcs properties were set correctly:
    [root@uppercaseovercloud-controller-0 keystone]# pcs property |grep rabbitmq
     uppercaseovercloud-controller-0: galera-role=true haproxy-role=true rabbitmq-role=true redis-role=true rmq-node-attr-last-known-rabbitmq=rabbit@uppercaseovercloud-controller-0
     uppercaseovercloud-controller-1: galera-role=true haproxy-role=true rabbitmq-role=true redis-role=true rmq-node-attr-last-known-rabbitmq=rabbit@uppercaseovercloud-controller-1
     uppercaseovercloud-controller-2: galera-role=true haproxy-role=true rabbitmq-role=true redis-role=true rmq-node-attr-last-known-rabbitmq=rabbit@uppercaseovercloud-controller-2

    Co-Authored-By: Damien Ciabrini <email address hidden>
    Depends-On: Ie240b8a4217827dd8ade82479a828817d63143ba
    Closes-bug: #1773219
    Change-Id: I5bd49c4a1b13b2310f8a1173aa6b86abfa5dab3d

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/571064

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (stable/queens)

Reviewed: https://review.openstack.org/571064
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=f90011dadbe2599ccf9ff4469bdea93413c6d0d2
Submitter: Zuul
Branch: stable/queens

commit f90011dadbe2599ccf9ff4469bdea93413c6d0d2
Author: Michele Baldessari <email address hidden>
Date: Thu May 24 18:47:02 2018 +0200

    Fix up property names in case of mixed case hostnames

    When deploying a stack that containes mixed-case hostnames
    the following error might be triggered:
    Debug: try 15/20: /usr/sbin/pcs -f
    /var/lib/pacemaker/cib/puppet-cib-backup20180405-8-1sqw3dc property set
    --node TEST-STACK34-controller-1 redis-role=true
    Debug: Error: Error: unable to set attribute redis-role
    Could not map name=TEST-STACK34-controller-1 to a UUID
    while the name in the cluster is test-stack34-controller-1

    This used to work pre-bundles because we used the facter provided
    $::hostname variable which was lower-cased for us. With bundles we
    switched to setting cluster properties from the service bootstrap nodes
    and so we used the '<service>_short_node_names' hiera key which might
    contain mixed-case hostnames.

    In order to fix this we just downcase() the short_node_names hiera
    string that we loop on so we can get the same behaviour we had on bare
    metal.

    Tested on an env with mixed-case hostnames:
    [root@uppercaseovercloud-controller-0 keystone]# hiera -c /etc/puppet/hiera.yaml rabbitmq_short_node_names
    ["UPPERCASEOverCloud-controller-0",
     "UPPERCASEOverCloud-controller-1",
     "UPPERCASEOverCloud-controller-2"]

    Cluster pcs properties were set correctly:
    [root@uppercaseovercloud-controller-0 keystone]# pcs property |grep rabbitmq
     uppercaseovercloud-controller-0: galera-role=true haproxy-role=true rabbitmq-role=true redis-role=true rmq-node-attr-last-known-rabbitmq=rabbit@uppercaseovercloud-controller-0
     uppercaseovercloud-controller-1: galera-role=true haproxy-role=true rabbitmq-role=true redis-role=true rmq-node-attr-last-known-rabbitmq=rabbit@uppercaseovercloud-controller-1
     uppercaseovercloud-controller-2: galera-role=true haproxy-role=true rabbitmq-role=true redis-role=true rmq-node-attr-last-known-rabbitmq=rabbit@uppercaseovercloud-controller-2

    Co-Authored-By: Damien Ciabrini <email address hidden>
    Depends-On: Ie240b8a4217827dd8ade82479a828817d63143ba
    Closes-bug: #1773219
    Change-Id: I5bd49c4a1b13b2310f8a1173aa6b86abfa5dab3d
    (cherry picked from commit f2484a0bf9e64ff84979abe9b70e7f65957b9d26)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/571714

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/571715

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (stable/pike)

Reviewed: https://review.openstack.org/571714
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=14f082d8e0903fe98f1a280ddf32a7f4fca553e4
Submitter: Zuul
Branch: stable/pike

commit 14f082d8e0903fe98f1a280ddf32a7f4fca553e4
Author: Michele Baldessari <email address hidden>
Date: Fri May 25 01:56:32 2018 +0200

    Lowercase any hostname comparisons in bootstrap_* scripts

    If you deploy a stack with mixed-case hostnames things will break
    because no task that is supposed to run on boostrap nodes will run
    due to the following code:
    HOSTNAME=$(/bin/hostname -s)
    SERVICE_NODEID=$(/bin/hiera -c /etc/puppet/hiera.yaml
    "${SERVICE_NAME}_short_bootstrap_node_name")
    if [[ "$HOSTNAME" == "$SERVICE_NODEID" ]]; then
    ...

    The hiera key might contain mixed-case letters whereas the hostname
    won't and the end result is going to be that no bootstrap tasks
    will run on any nodes and, amongst other things, no database tables
    will be created, making all services unusable.

    Since we use bash explicitely we can leverage the ${var,,} expression
    for this.

    Change-Id: Ie240b8a4217827dd8ade82479a828817d63143ba
    Related-Bug: #1773219
    (cherry picked from commit 3345557d7b64b29ccee258de8bcd6cfbfa88291e)
    (cherry picked from commit 57dea6a3c77456b96d84d3e4f3d2881424194765)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (stable/pike)

Reviewed: https://review.openstack.org/571715
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=d890ec5237cfdf4ffb48430988b6554f809224b5
Submitter: Zuul
Branch: stable/pike

commit d890ec5237cfdf4ffb48430988b6554f809224b5
Author: Michele Baldessari <email address hidden>
Date: Thu May 24 18:47:02 2018 +0200

    Fix up property names in case of mixed case hostnames

    When deploying a stack that containes mixed-case hostnames
    the following error might be triggered:
    Debug: try 15/20: /usr/sbin/pcs -f
    /var/lib/pacemaker/cib/puppet-cib-backup20180405-8-1sqw3dc property set
    --node TEST-STACK34-controller-1 redis-role=true
    Debug: Error: Error: unable to set attribute redis-role
    Could not map name=TEST-STACK34-controller-1 to a UUID
    while the name in the cluster is test-stack34-controller-1

    This used to work pre-bundles because we used the facter provided
    $::hostname variable which was lower-cased for us. With bundles we
    switched to setting cluster properties from the service bootstrap nodes
    and so we used the '<service>_short_node_names' hiera key which might
    contain mixed-case hostnames.

    In order to fix this we just downcase() the short_node_names hiera
    string that we loop on so we can get the same behaviour we had on bare
    metal.

    Tested on an env with mixed-case hostnames:
    [root@uppercaseovercloud-controller-0 keystone]# hiera -c /etc/puppet/hiera.yaml rabbitmq_short_node_names
    ["UPPERCASEOverCloud-controller-0",
     "UPPERCASEOverCloud-controller-1",
     "UPPERCASEOverCloud-controller-2"]

    Cluster pcs properties were set correctly:
    [root@uppercaseovercloud-controller-0 keystone]# pcs property |grep rabbitmq
     uppercaseovercloud-controller-0: galera-role=true haproxy-role=true rabbitmq-role=true redis-role=true rmq-node-attr-last-known-rabbitmq=rabbit@uppercaseovercloud-controller-0
     uppercaseovercloud-controller-1: galera-role=true haproxy-role=true rabbitmq-role=true redis-role=true rmq-node-attr-last-known-rabbitmq=rabbit@uppercaseovercloud-controller-1
     uppercaseovercloud-controller-2: galera-role=true haproxy-role=true rabbitmq-role=true redis-role=true rmq-node-attr-last-known-rabbitmq=rabbit@uppercaseovercloud-controller-2

    Co-Authored-By: Damien Ciabrini <email address hidden>
    Depends-On: Ie240b8a4217827dd8ade82479a828817d63143ba
    Closes-bug: #1773219
    Change-Id: I5bd49c4a1b13b2310f8a1173aa6b86abfa5dab3d
    (cherry picked from commit f2484a0bf9e64ff84979abe9b70e7f65957b9d26)
    (cherry picked from commit f90011dadbe2599ccf9ff4469bdea93413c6d0d2)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/puppet-tripleo 9.1.0

This issue was fixed in the openstack/puppet-tripleo 9.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/puppet-tripleo 8.3.4

This issue was fixed in the openstack/puppet-tripleo 8.3.4 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/puppet-tripleo 7.4.14

This issue was fixed in the openstack/puppet-tripleo 7.4.14 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.