periodic-tripleo-centos-7-master-containers-build fails for mariadb:Error'd /bin/sh -c rm /etc/my.cnf.d/mariadb-server.cnf /etc/my.cnf.d/auth_gssapi.cnf HTTP Error 503 - Service Unavailable

[Marios Andreou]

The periodic-tripleo-centos-7-master-containers-build is failing during the TASK [rdo-kolla-build : Build and push images with 503 errors , trace from [1] like:

  bc140c8149fc43a5248fccff0daeef38182e49f6fe75d9b46db1206dc25a6c1c-c7-x86_64-comps.xml.gz: [Errno 14] HTTP Error 503 - Service Unavailable\nINFO:kolla.common.utils.horizon:Trying other mirror.\nINFO:kolla.common.utils.horizon:\u001b[0m\nINFO:kolla.common.utils.mariadb:\u001b[91mrm: cannot remove '/etc/my.cnf.d/auth_gssapi.cnf': No such file or directory\nINFO:kolla.common.utils.mariadb:\u001b[0m\nINFO:kolla.common.utils.mariadb:Removing intermediate container a9dc4b5e7cab\nERROR:kolla.common.utils.mariadb:Error'd with the following message\nERROR:kolla.common.utils.mariadb:The command '/bin/sh -c rm /etc/my.cnf.d/mariadb-server.cnf /etc/my.cnf.d/auth_gssapi.cnf' returned a non-zero code: 1\nINFO:kolla.common.utils:Attempt number: 3 to run task: BuildTask(mariadb) \nINFO:kolla.common.utils.mariadb:Building\nINFO:kolla.common.utils.ceilometer-base:\u001b[91mhttp://mirror.san.fastserv.com/pub/linux/centos/7.6.1810/os/x86_64/repodata/bc140c8149fc43a5248fccff0daeef38182e49f6fe75d9b46db1206dc25a6c1c-c7-x86_64-comps.xml.gz: [Errno 14] HTTP Error 503 - Service Unavailable\
...
  mirror.san.fastserv.com\nINFO:kolla.common.utils.sahara-base: * updates: mirror.dal10.us.leaseweb.net\nINFO:kolla.common.utils.nova-base: * base: mirror.san.fastserv.com\nINFO:kolla.common.utils.nova-base: * extras: mirror.san.fastserv.com\nINFO:kolla.common.utils.nova-base: * updates: mirrors.xmission.com\nINFO:kolla.common.utils.octavia-base: * base: mirror.san.fastserv.com\nINFO:kolla.common.utils.octavia-base: * extras: mirror.san.fastserv.com\nINFO:kolla.common.utils.octavia-base: * updates: mirrors.ocf.berkeley.edu\nINFO:kolla.common.utils.nova-libvirt: \nINFO:kolla.common.utils.nova-libvirt: Installing : python-oslo-utils-lang-3.37.1-0.20181012100734.6e0b90b 179/391\nINFO:kolla.common.utils.nova-libvirt: \nINFO:kolla.common.utils.nova-libvirt: Installing : python2-jmespath-0.9.0-5.el7.noarch 180/391\nINFO:kolla.common.utils.nova-libvirt: \nINFO:kolla.common.utils.nova-libvirt: Installing : 1:quota-nls-4.01-17.el7.noarch 181/391\nINFO:kolla.common.utils.nova-libvirt: \nINFO:kolla.common.utils.nova-libvirt: Installing : python2-pysocks-1.5.6-3.el7.noarch 182/391\nINFO:kolla.common.utils.nova-libvirt: \nINFO:kolla.common.utils.nova-libvirt: Installing : 40:libcacard-2.5.2-2.1.el7.x86_64 183/391\nINFO:kolla.common.utils.nova-libvirt: \nINFO:kolla.common.utils.nova-libvirt: Installing : python2-requestsexceptions-1.3.0-0.20180129172822.d7ac 184/391\nINFO:kolla.common.utils.nova-libvirt: \nINFO:kolla.common.utils.nova-libvirt: Installing : grubby-8.28-25.el7.x86_64 185/391\nINFO:kolla.common.utils.octavia-base:\u001b[91mhttp://mirror.san.fastserv.com/pub/linux/centos/7.6.1810/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable\
  file the bug

[1] http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/06f5cb2/job-output.txt.gz

[Damien Ciabrini (dciabrin)]

This is because we broke kolla image rebuild when we pushed mariadb 10.3.10 in Stein [1]

A fix [2] has been created yesterday to support building mariadb image with both 10.1 and 10.3, and we now have periodic-tripleo-centos-7-master-containers-build passing [3] when using that review.

So that error should disappear once [2] is merged.

[1] https://review.rdoproject.org/r/#/c/16911 , https://review.rdoproject.org/r/#/c/17707/
[2] https://review.openstack.org/#/c/624063/
[3] https://review.rdoproject.org/r/#/c/17730/

[Marios Andreou (marios-b)]

green run today [1] periodic-tripleo-centos-7-master-containers-build via http://cistatus.tripleo.org/promotion/

so finally had new consistent build looks like

[1] http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/9427359/

[Damien Ciabrini (dciabrin)]

Yes! I can see the fix in the Dockerfile action [1].

Marios, does that mean that the newly built containers are now available for consumption in CI jobs, or are there other steps pending until their promotion?

[1] https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/9427359/logs/kolla/logs/mariadb.log

[Marios Andreou (marios-b)]

well the new containers will become available as the new current-tripleo upon master promotion which we haven't had since 4th Dec. The containers-build job here was blocking that promotion. All the promotion jobs/stages are periodic so we can only wait (and clear any other blockers if they come up/we find them).

The caveat here is that the thing being blocked was a promotion job which itself is then blocking the master promotion. For regular check/gate jobs we aren't seeing this fail because they are using master tripleo-common.

some info here about current-tripleo and containers after promotion see "Upstream Promotion Pipeline" https://docs.openstack.org/tripleo-docs/latest/contributor/stages-overview.html
hope it helps?