Master tripleo component is red

[Cédric Jeanneret]

It seems a change related to HAProxy is causing issues in the tripleo component:
http://dashboard-ci.tripleo.org/d/WWV51gO4k/component-pipeline?orgId=1&from=now-2d&to=now&viewPanel=483

https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-tripleo-master/2d87453/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

WARNING | ERROR: Can't run container container-puppet-haproxy
stderr: time="2023-01-24T20:02:59Z" level=info msg="podman filtering at log level info"
time="2023-01-24T20:02:59Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
time="2023-01-24T20:02:59Z" level=info msg="Setting parallel job count to 25"
time="2023-01-24T20:03:00Z" level=info msg="Sysctl net.ipv4.ping_group_range=0 0 ignored in containers.conf, since Network Namespace set to host"
time="2023-01-24T20:03:00Z" level=info msg="User mount overriding libpod mount at \"/etc/hosts\""
time="2023-01-24T20:03:00Z" level=info msg="Running conmon under slice machine.slice and unitName libpod-conmon-c691f384f8d930e1805b6036c74ed5ebee2772a7384576760e9b70dbdaf25f4c.scope"
time="2023-01-24T20:03:00Z" level=info msg="Got Conmon PID as 118929"
time="2023-01-24T20:03:00Z" level=info msg="Received shutdown.Stop(), terminating!" PID=118801
+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,haproxy_config"' /etc/config.pp
+ logger -s -t puppet-user
<13>Jan 24 20:03:00 puppet-user: Warning: Facter: Could not delete cache: Read-only file system @ apply2files - /opt/puppetlabs/facter/cache/cached_facts/operating system
<13>Jan 24 20:03:04 puppet-user: Warning: Facter: Could not delete cache: Read-only file system @ apply2files - /opt/puppetlabs/facter/cache/cached_facts/operating system
<13>Jan 24 20:03:04 puppet-user: Warning: Facter: Could not delete cache: Read-only file system @ apply2files - /opt/puppetlabs/facter/cache/cached_facts/operating system
<13>Jan 24 20:03:04 puppet-user: Warning: Facter: Could not delete cache: Read-only file system @ apply2files - /opt/puppetlabs/facter/cache/cached_facts/operating system
<13>Jan 24 20:03:04 puppet-user: Warning: Facter: Could not delete cache: Read-only file system @ apply2files - /opt/puppetlabs/facter/cache/cached_facts/operating system
<13>Jan 24 20:03:04 puppet-user: Warning: Facter: Could not delete cache: Read-only file system @ apply2files - /opt/puppetlabs/facter/cache/cached_facts/networking
<13>Jan 24 20:03:04 puppet-user: Warning: Facter: Could not delete cache: Read-only file system @ apply2files - /opt/puppetlabs/facter/cache/cached_facts/operating system
<13>Jan 24 20:03:04 puppet-user: Warning: Facter: Could not delete cache: Read-only file system @ apply2files - /opt/puppetlabs/facter/cache/cached_facts/operating system
<13>Jan 24 20:03:06 puppet-user: Warning: Facter: Could not delete cache: Read-only file system @ apply2files - /opt/puppetlabs/facter/cache/cached_facts/operating system
<13>Jan 24 20:03:06 puppet-user: Warning: Facter: Could not delete cache: Read-only file system @ apply2files - /opt/puppetlabs/facter/cache/cached_facts/operating system
<13>Jan 24 20:03:06 puppet-user: Warning: Facter: Could not delete cache: Read-only file system @ apply2files - /opt/puppetlabs/facter/cache/cached_facts/operating system
<13>Jan 24 20:03:06 puppet-user: Warning: Facter: Could not delete cache: Read-only file system @ apply2files - /opt/puppetlabs/facter/cache/cached_facts/operating system
<13>Jan 24 20:03:06 puppet-user: Warning: Facter: Could not delete cache: Read-only file system @ apply2files - /opt/puppetlabs/facter/cache/cached_facts/operating system
<13>Jan 24 20:03:06 puppet-user: Warning: Facter: Could not delete cache: Read-only file system @ apply2files - /opt/puppetlabs/facter/cache/cached_facts/operating system
<13>Jan 24 20:03:07 puppet-user: Warning: Found multiple default providers for service: pacemaker_xml, swiftinit, base; using pacemaker_xml
<13>Jan 24 20:03:09 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
<13>Jan 24 20:03:09 puppet-user: (file: /etc/puppet/hiera.yaml)
<13>Jan 24 20:03:09 puppet-user: Warning: Undefined variable '::deploy_config_name';
<13>Jan 24 20:03:09 puppet-user: (file & line not available)
<13>Jan 24 20:03:10 puppet-user: Warning: Scope(Haproxy::Frontend[haproxy.stats]): The $bind_options parameter is deprecated; please use $bind instead
<13>Jan 24 20:03:10 puppet-user: Warning: Scope(Haproxy::Frontend[horizon]): The $bind_options parameter is deprecated; please use $bind instead
<13>Jan 24 20:03:10 puppet-user: Warning: Scope(Haproxy::Backend[horizon_be]): Overriding the value of $sort_options_alphabetic to "false" due to "httpchk" option defined
<13>Jan 24 20:03:10 puppet-user: Warning: Scope(Haproxy::Frontend[mysql]): The $bind_options parameter is deprecated; please use $bind instead
<13>Jan 24 20:03:10 puppet-user: Warning: Scope(Haproxy::Backend[mysql_be]): Overriding the value of $sort_options_alphabetic to "false" due to "httpchk" option defined
<13>Jan 24 20:03:10 puppet-user: Warning: Unknown variable: '::pacemaker::pcs_010'. (file: /etc/puppet/modules/pacemaker/manifests/resource/bundle.pp, line: 159, column: 6)
<13>Jan 24 20:03:10 puppet-user: Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.
<13>Jan 24 20:03:10 puppet-user: Warning: You cannot collect exported resources without storeconfigs being set; the collection will be ignored (file: /etc/puppet/modules/haproxy/manifests/balancermember/collect_exported.pp, line: 7, column: 3)
<13>Jan 24 20:03:10 puppet-user: Warning: Scope(Haproxy::Frontend[keystone_admin]): The $bind_options parameter is deprecated; please use $bind instead
<13>Jan 24 20:03:10 puppet-user: Error: Evaluation Error: Error while evaluating a Resource Statement, Haproxy::Balancermember[keystone_admin]: parameter 'ports' expects a value of type Undef, Array, or String, got Integer (file: /etc/puppet/modules/tripleo/manifests/haproxy/endpoint.pp, line: 313) on node standalone.localdomain
+ rc=1
+ '[' false = false ']'
+ set +x
2023-01-24 15:03:48.839506 | fa163e7c-7b99-8b18-cf43-000000002c75 | FATAL | Manage Puppet containers (generate config) for step 1 | standalone | error={"changed": false, "msg": "Failed containers: container-puppet-haproxy"}
2023-01-24 15:03:48.839994 | fa163e7c-7b99-8b18-cf43-000000002c75 | TIMING | Manage Puppet containers (generate config) for step 1 | standalone | 0:10:20.503482 | 49.60s

[Luca Miccini (lmiccini2)]

this seems to be related to https://github.com/puppetlabs/puppetlabs-haproxy/commit/95ca912f77966ed0da7a5bec9265cdacbf09da4c

puppet-haproxy-6.4.1-0.20230124153846.95ca912.el9.noarch picked up this change.

I am testing this workaround:

"dnf downgrade -y puppet-haproxy"

that rolls back to:

6.4.1-0.20230118145422.9fbf4e6.el9

We should probably pin this for the time being.

[Cédric Jeanneret (cjeanner)]

Temporary pinning for master: https://review.rdoproject.org/r/c/rdoinfo/+/46709

That will give time for a proper fix in puppet-tripleo

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/puppet-tripleo/+/871743

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/puppet-tripleo/+/871807

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Takashi Kajinami <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/puppet-tripleo/+/871743
Reason: wrong branch. will resubmit this to master.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-tripleo/+/871807
Committed: https://opendev.org/openstack/puppet-tripleo/commit/6372a182e4366c9c49c69d80bf2418ff0e7f5e58
Submitter: "Zuul (22348)"
Branch: master

commit 6372a182e4366c9c49c69d80bf2418ff0e7f5e58
Author: Takashi Kajinami <email address hidden>
Date: Thu Jan 26 01:45:56 2023 +0900

    Ensure haproxy::balancemember::ports is a string

    The recent change[1] in puppetlabs-haproxy introduced data type check
    and now the ports parameter accept only string or array of strings and
    no longer accepts an integer.

    This ensures the value is converted to avoid validation errors.

    [1] https://github.com/puppetlabs/puppetlabs-haproxy/commit/95ca912f77966ed0da7a5bec9265cdacbf09da4c

    Closes-Bug: #2003882
    Change-Id: I65720576249f2242331e38762dd0d5860d74de9d

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 18.0.0 release.