uc installation fails with parameter 'ssl_cacert' expects a Stdlib::Absolutepath

Bug #1947776 reported by Michele Baldessari
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Michele Baldessari

Bug Description

Lon hit this issue and I can reproduce it:
2021-10-19 18:12:02.562474 | | WARNING | ERROR: Can't run container container-puppet-rabbitmq
stderr: + /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron"' /etc/config.pp
+ logger -s -t puppet-user
<13>Oct 19 18:11:57 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
<13>Oct 19 18:12:02 puppet-user: (file: /etc/puppet/hiera.yaml)
<13>Oct 19 18:12:02 puppet-user: Warning: Undefined variable '::deploy_config_name';
<13>Oct 19 18:12:02 puppet-user: (file & line not available)
<13>Oct 19 18:12:02 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.10/deprecated_language.html
<13>Oct 19 18:12:02 puppet-user: (file & line not available)
<13>Oct 19 18:12:02 puppet-user: Error: Evaluation Error: Error while evaluating a Resource Statement, Class[Rabbitmq]:
<13>Oct 19 18:12:02 puppet-user: parameter 'ssl_cacert' expects a Stdlib::Absolutepath = Variant[Stdlib::Windowspath = Pattern[/\A(([a-zA-Z]:[\\\/])|([\\\/][\\\/][^\\\/]+[\\\/][^\\\/]+)|([\\\/][\\\/]\?[\\\/][^\\
\/]+)).*\z/], Stdlib::Unixpath = Pattern[/\A\/([^\n\/\0]+\/*)*\z/]] value, got String
<13>Oct 19 18:12:02 puppet-user: parameter 'ssl_management_cacert' expects a Stdlib::Absolutepath = Variant[Stdlib::Windowspath = Pattern[/\A(([a-zA-Z]:[\\\/])|([\\\/][\\\/][^\\\/]+[\\\/][^\\\/]+)|([\\\/][\\\/]\
?[\\\/][^\\\/]+)).*\z/], Stdlib::Unixpath = Pattern[/\A\/([^\n\/\0]+\/*)*\z/]] value, got String (file: /etc/puppet/modules/tripleo/manifests/profile/base/rabbitmq.pp, line: 243, column: 7) on node undercloud-0.ho
me.arpa
+ rc=1
+ '[' false = false ']'
+ set +x
2021-10-19 18:12:02.563524 | 52540085-7238-d687-4d49-0000000010cd | FATAL | Manage Puppet containers (generate config) for step 1 | undercloud-0 | error={"changed": false, "msg": "Failed containers: container
-puppet-rabbitmq"}
2021-10-19 18:12:02.564039 | 52540085-7238-d687-4d49-0000000010cd | TIMING | Manage Puppet containers (generate config) for step 1 | undercloud-0 | 0:03:58.858023 | 15.44s

The undercloud.conf that triggers this issue is:
[DEFAULT]
overcloud_domain_name = home.arpa
undercloud_hostname = undercloud-0.home.arpa
undercloud_ntp_servers=clock1.rdu2.redhat.com
undercloud_timezone = UTC
container_images_file=/home/stack/containers-prepare-parameter.yaml
undercloud_service_certificate = /etc/pki/instack-certs/undercloud.pem
local_interface = enp2s0
local_ip = 192.168.24.1/24
undercloud_public_host = 192.168.24.2
undercloud_admin_host = 192.168.24.3
subnets = ctlplane-subnet
local_subnet = ctlplane-subnet
[ctlplane-subnet]
local_subnet = ctlplane-subnet
cidr = 192.168.24.0/24
dhcp_start = 192.168.24.160
dhcp_end = 192.168.24.170
gateway = 192.168.24.1
inspection_iprange = 192.168.24.171,192.168.24.175
masquerade = true

Revision history for this message
Michele Baldessari (michele) wrote :

The issue is that v1/undercloud_config.py sets the following:
env_data['InternalTLSCAFile'] = ''

This then gets passed to the rabbitmq.yaml file:
rabbitmq::ssl_cacert: {get_param: InternalTLSCAFile}
rabbitmq::ssl_management_cacert: {get_param: InternalTLSCAFile}

and so we have the the following hiera keys:
/etc/puppet/hieradata/service_configs.json: "rabbitmq::ssl": false,
/etc/puppet/hieradata/service_configs.json: "rabbitmq::ssl_cacert": "",
/etc/puppet/hieradata/service_configs.json: "rabbitmq::ssl_depth": 1,
/etc/puppet/hieradata/service_configs.json: "rabbitmq::ssl_erl_dist": false,
/etc/puppet/hieradata/service_configs.json: "rabbitmq::ssl_interface": "%{hiera('ctlplane')}",
/etc/puppet/hieradata/service_configs.json: "rabbitmq::ssl_management_cacert": "",
/etc/puppet/hieradata/service_configs.json: "rabbitmq::ssl_only": false,

Which then confuses the puppet-rabbitmq module. So fundamentally we always pushed a configuration that only now fails since we started using the proper ca, because the empty cacert does not validate in puppet-rabbitmq.

Revision history for this message
Michele Baldessari (michele) wrote :

Reverting https://review.opendev.org/c/openstack/python-tripleoclient/+/588246/3/tripleoclient/v1/undercloud_config.py fixes it, but I'll try something in puppet-tripleo as I am afraid of regressing there otherwise.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to puppet-tripleo (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/puppet-tripleo/+/814638

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to puppet-tripleo (master)

Reviewed: https://review.opendev.org/c/openstack/puppet-tripleo/+/814638
Committed: https://opendev.org/openstack/puppet-tripleo/commit/a9e0ec90f493880a8783b1654c82a4578df2a458
Submitter: "Zuul (22348)"
Branch: master

commit a9e0ec90f493880a8783b1654c82a4578df2a458
Author: Michele Baldessari <email address hidden>
Date: Tue Oct 19 21:33:36 2021 +0200

    Make sure that rabbitmq_cacert has 'undef' as a default.

    This is done so that the deployment does not fail if we do not pass any
    rabbitmq:ssl_cacert key at all. This is needed on SSL (non TLS-E)
    undercloud installations, see LP and related tht patch.

    Change-Id: I4e9801ba3ced734e5aa3fa22df522fab0e84761a
    Related-Bug: #1947776

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/814639
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/f8e7bf2bba79120fcabcb60d889333df06a3bc8a
Submitter: "Zuul (22348)"
Branch: master

commit f8e7bf2bba79120fcabcb60d889333df06a3bc8a
Author: Michele Baldessari <email address hidden>
Date: Tue Oct 19 21:38:13 2021 +0200

    Do not set rabbitmq SSL CA certs when InternalTLSCAFile is ''

    The undercloud installation sets InternalTLSCAFile to '' when
    undercloud_service_certificate is set (done via
    I5d7f35194f98b2d5c06a417cac75d52ff646def0 "undercloud: Disable CA path
    if user-provided cert is used"). In this case we end up setting
    rabbitmq::{ssl_cacert,ssl_management_cacert} to '' which then breaks
    puppet-rabbitmq because it verifies that they are absolute paths.

    Let's just skip passing them when InternalTLSCAFile is set to
    empty.

    Depends-On: I4e9801ba3ced734e5aa3fa22df522fab0e84761a
    Closes-Bug: #1947776
    Change-Id: I16f0fd04e1ae941d09d567eadaeb5bda959099fc

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to puppet-tripleo (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/puppet-tripleo/+/814757

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/814824

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to puppet-tripleo (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/puppet-tripleo/+/814757
Committed: https://opendev.org/openstack/puppet-tripleo/commit/3f03b8a54933a2293faf823b5b979a4bd6e3be02
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 3f03b8a54933a2293faf823b5b979a4bd6e3be02
Author: Michele Baldessari <email address hidden>
Date: Tue Oct 19 21:33:36 2021 +0200

    Make sure that rabbitmq_cacert has 'undef' as a default.

    This is done so that the deployment does not fail if we do not pass any
    rabbitmq:ssl_cacert key at all. This is needed on SSL (non TLS-E)
    undercloud installations, see LP and related tht patch.

    Change-Id: I4e9801ba3ced734e5aa3fa22df522fab0e84761a
    Related-Bug: #1947776
    (cherry picked from commit a9e0ec90f493880a8783b1654c82a4578df2a458)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/814824
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/35ab8cde378f8044a243ffbdca8fa14ebddaecd3
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 35ab8cde378f8044a243ffbdca8fa14ebddaecd3
Author: Michele Baldessari <email address hidden>
Date: Tue Oct 19 21:38:13 2021 +0200

    Do not set rabbitmq SSL CA certs when InternalTLSCAFile is ''

    The undercloud installation sets InternalTLSCAFile to '' when
    undercloud_service_certificate is set (done via
    I5d7f35194f98b2d5c06a417cac75d52ff646def0 "undercloud: Disable CA path
    if user-provided cert is used"). In this case we end up setting
    rabbitmq::{ssl_cacert,ssl_management_cacert} to '' which then breaks
    puppet-rabbitmq because it verifies that they are absolute paths.

    Let's just skip passing them when InternalTLSCAFile is set to
    empty.

    NB: Slight context conflict around key_size_override_unset
        and around postsave_cmd

    Depends-On: I4e9801ba3ced734e5aa3fa22df522fab0e84761a
    Closes-Bug: #1947776
    Change-Id: I16f0fd04e1ae941d09d567eadaeb5bda959099fc

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 16.0.0

This issue was fixed in the openstack/tripleo-heat-templates 16.0.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.