[CI] Upgrade job broken by attempting to use C.UTF-8 locale

[Jiří Stránský]

Error: Failed to apply catalog: Cannot parse invalid user line: /etc/rabbitmq/rabbitmq-env.conf: line 2: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8): No such file or directory

It says just warning but looks suspiciously like it could actually be the cause of the error...

[Jiří Stránský (jistr)]

Broken job noticed at:

https://review.openstack.org/#/c/625890/

Patches which could potentially be related to the breakage:

https://review.openstack.org/#/c/649170 (passed the job)
https://review.openstack.org/#/c/649299 (failed the job)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/649696

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/649697

[Michele Baldessari (michele)]

* RHEL8 containers have no en_US.UTF-8 locale only C.UTF-8 :
 [root@undercloud-0 ~]# podman run -it --rm --net=host "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-rabbitmq:latest" sh
()[rabbitmq@undercloud-0 /]$ ls -l /usr/lib/locale/
total 0
drwxr-xr-x. 3 root root 227 Mar 13 18:26 C.utf8
()[rabbitmq@undercloud-0 /]$ export LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
()[rabbitmq@undercloud-0 /]$ export LANG=C.UTF-8 LC_ALL=C.UTF-8
()[rabbitmq@undercloud-0 /]$

[root@undercloud-0 ~]# podman run -it --rm --net=host "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-rabbitmq:latest" sh -c 'l
ocale -a'
C
C.utf8
POSIX

[root@undercloud-0 ~]# podman run -it --rm --net=host "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-rabbitmq:latest" sh -c '
rpm -qa |grep glibc'
glibc-2.28-42.el8.x86_64
glibc-common-2.28-42.el8.x86_64
glibc-minimal-langpack-2.28-42.el8.x86_64

* RHEL7/Centos7 containers have no C.UTF-8 locale and only en_US.UTF-8 locale:
 [root@undercloud-0 ~]# podman run -it --rm --net=host "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp14/openstack-rabbitmq:latest" sh
()[rabbitmq@undercloud-0 /]$ export LANG=C.UTF-8 LC_ALL=C.UTF-8
sh: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
sh: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
()[rabbitmq@undercloud-0 /]$ export LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
()[rabbitmq@undercloud-0 /]$

 [root@undercloud-0 ~]# podman run -it --rm --net=host "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp14/openstack-rabbitmq:latest" sh -c '
locale -a'
C
POSIX
en_US.utf8

 [root@undercloud-0 ~]# podman run -it --rm --net=host "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp14/openstack-rabbitmq:latest" sh -c
'rpm -qa |grep glibc'
glibc-2.17-260.el7_6.3.x86_64
glibc-common-2.17-260.el7_6.3.x86_64

Three possible solutions:
1) We parametrize the rabbitmq_container_locale and set it to en_US.UTF-8 in upstream and C.UTF-8 downstream.
When centos 8 is out and we base all the things on it we set the default to C.UTF-8 and done.

2) We leave it at C.UTF-8 but we make sure that the upstream centos 7 rabbitmq containers have this locale

3) We carry two downstream patches only that set the rabbitmq container locale to C.UTF-8 while upstream we
leave en_US.UTF-8 until centos8 comes out and CI consumes it.

[Michele Baldessari (michele)]

Option 2) is a no go as rhel7 does not have C.UTF-8

The only sensible option I think is to move hardcode en_US.UTF-8 upstream and make sure we install glibc-langpack-en-2.28-42.el8.x86_64 downstream only so that it will work inside the rabbitmq container. We tested this and it works okay.

[Michele Baldessari (michele)]

So the plan is:
A) merge https://review.openstack.org/#/c/649697/ and https://review.openstack.org/#/c/649696/ so we get en_US.UTF-8 as a default
B) push a downstream only patch that installs glibc-langpack-en-2.28-42.el8.x86_64 in the rabbitmq rhel8 containers

[Michele Baldessari (michele)]

No downstream change but we will pull glibc-langpack-en in via puppet-tripleo rpm dep conditionalized on rhel8 https://review.rdoproject.org/r/#/c/19963/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.openstack.org/649695

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/649697
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ceb005b07f84deb491afa19238833edc06c06531
Submitter: Zuul
Branch: master

commit ceb005b07f84deb491afa19238833edc06c06531
Author: Jiri Stransky <email address hidden>
Date: Wed Apr 3 20:56:39 2019 +0200

    Fix RabbitMQ locale for CentOS 7 (t-h-t part)

    It seems that CentOS 7 does not have C.UTF-8 locale. Since we need
    UTF-8-based locale, use en_US.UTF-8 instead.

    Change-Id: Id510565edd33e6f4d409afc7a4afb938ab2f9fd0
    Partial-Bug: #1823062

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/649696
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=566703dc27d11b6aef6c41ecebc4d2e4110d6cc5
Submitter: Zuul
Branch: master

commit 566703dc27d11b6aef6c41ecebc4d2e4110d6cc5
Author: Jiri Stransky <email address hidden>
Date: Wed Apr 3 20:53:01 2019 +0200

    Fix RabbitMQ locale for CentOS 7 (Puppet part)

    It seems that CentOS 7 does not have C.UTF-8 locale. Since we need
    UTF-8-based locale, use en_US.UTF-8 instead.

    Change-Id: I25d2b9a227a7c5de127bdfd9d2f387be9eea01e0
    Partial-Bug: #1823062