container-puppet.py isn't regenerated during update.

Bug #1853156 reported by Sofer Athlan-Guyot
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Sofer Athlan-Guyot

Bug Description

Hi,

discovered there https://bugzilla.redhat.com/show_bug.cgi?id=1774076

We needed a fix for the json vs k8s driver usage in podman to be delivered in container-puppet.py on the overcloud.

It happens that file isn't regenerated during update step while we need to generate the puppet configuration during update, ie trigger some podman start/run command.

This make the update fails with this kind of error:

2019-11-15 05:29:44 | "2019-11-15 05:28:10,139 ERROR: 178645 -- ['/usr/bin/podman', 'run', '--user', 'root', '--name', 'container-puppet-haproxy', '--env', 'PUPPET_TAGS=file,file_line,concat,augeas,cron,ha
proxy_config', '--env', 'NAME=haproxy', '--env', 'HOSTNAME=controller-0', '--env', 'NO_ARCHIVE=', '--env', 'STEP=6', '--env', 'NET_HOST=true', '--log-driver', 'json-file', '--volume', '/etc/localtime:/etc/localtim
e:ro', '--volume', '/tmp/tmpkkpdgv7j:/etc/config.pp:ro', '--volume', '/etc/puppet/:/tmp/puppet-etc/:ro', '--volume', '/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume', '/etc/pki/tls/certs/ca
-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume', '/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--vo
lume', '/var/lib/config-data:/var/lib/config-data/:rw', '--volume', '/var/lib/container-puppet/puppetlabs/facter.conf:/etc/puppetlabs/facter/facter.conf:ro', '--volume', '/var/lib/container-puppet/puppetlabs/:/opt
/puppetlabs/:ro', '--volume', '/dev/log:/dev/log:rw', '--log-opt', 'path=/var/log/containers/stdouts/container-puppet-haproxy.log', '--security-opt', 'label=disable', '--volume', '/usr/share/openstack-puppet/modul
es/:/usr/share/openstack-puppet/modules/:ro', '--volume', '/etc/pki/tls/private/overcloud_endpoint.pem:/etc/pki/tls/private/overcloud_endpoint.pem:ro', '--entrypoint', '/var/lib/container-puppet/container-puppet.s
h', '--net', 'host', '--volume', '/etc/hosts:/etc/hosts:ro', '--volume', '/var/lib/container-puppet/container-puppet.sh:/var/lib/container-puppet/container-puppet.sh:ro', '192.168.24.1:8787/rh-osbs/rhosp15-opensta
ck-haproxy:20191106.1'] run failed after [conmon:e]: No such log driver json-file",
2019-11-15 05:29:44 | "Error: write child: broken pipe",
2019-11-15 05:29:44 | " attempt(s): 1",
2019-11-15 05:29:44 | "2019-11-15 05:28:10,140 WARNING: 178645 -- Retrying running container: haproxy",
2019-11-15 05:29:44 | "2019-11-15 05:28:16,325 ERROR: 178645 -- ['/usr/bin/podman', 'start', '-a', 'container-puppet-haproxy'] run failed after [conmon:e]: No such log driver json-file",
2019-11-15 05:29:44 | "Error: unable to start container faae0f9449ae1fb795511196ceb47b3b7ea04b357d95088dbecde8508bf6622b: write child: broken pipe",
2019-11-15 05:29:44 | " attempt(s): 2",

Changed in tripleo:
assignee: nobody → Sofer Athlan-Guyot (sofer-athlan-guyot)
Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
Sofer Athlan-Guyot (sofer-athlan-guyot) wrote :
tags: added: stein-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/695654

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/695039
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=10a523a03d6c6883015e620de52c64b85d20a188
Submitter: Zuul
Branch: master

commit 10a523a03d6c6883015e620de52c64b85d20a188
Author: Sofer Athlan-Guyot <email address hidden>
Date: Tue Nov 19 18:06:28 2019 +0100

    Make sure we apply all deploy step-0 during update.

    That step include, in particular, the puppet-container.py/sh creation
    file plus some other task that should happen very early during the
    deploy process.

    During update if a change in puppet-container.py happen in the
    template we have to trigger those steps to get the changes. This is
    especially critical because we run puppet container configuration
    during the update stage.

    Closes-Bug: #1853156
    Change-Id: I26406da82c584dc5093c17ad26f263057a5cbcaa

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/train)

Reviewed: https://review.opendev.org/695654
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=3c71c99d7c843ac87616621a4eb084b0852b65f8
Submitter: Zuul
Branch: stable/train

commit 3c71c99d7c843ac87616621a4eb084b0852b65f8
Author: Sofer Athlan-Guyot <email address hidden>
Date: Tue Nov 19 18:06:28 2019 +0100

    Make sure we apply all deploy step-0 during update.

    That step include, in particular, the puppet-container.py/sh creation
    file plus some other task that should happen very early during the
    deploy process.

    During update if a change in puppet-container.py happen in the
    template we have to trigger those steps to get the changes. This is
    especially critical because we run puppet container configuration
    during the update stage.

    Closes-Bug: #1853156
    Change-Id: I26406da82c584dc5093c17ad26f263057a5cbcaa
    (cherry picked from commit 10a523a03d6c6883015e620de52c64b85d20a188)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/695861

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/695861
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=886c51549f06976a44ceed3ab0963b87afa4d79b
Submitter: Zuul
Branch: stable/stein

commit 886c51549f06976a44ceed3ab0963b87afa4d79b
Author: Sofer Athlan-Guyot <email address hidden>
Date: Tue Nov 19 18:06:28 2019 +0100

    Make sure we apply all deploy step-0 during update.

    That step include, in particular, the puppet-container.py/sh creation
    file plus some other task that should happen very early during the
    deploy process.

    During update if a change in puppet-container.py happen in the
    template we have to trigger those steps to get the changes. This is
    especially critical because we run puppet container configuration
    during the update stage.

    Patch slighly adjusted for the backport as step_1 refactoring doesn't
    exist in stein and neither does container-puppet.sh.

    Closes-Bug: #1853156
    Change-Id: I26406da82c584dc5093c17ad26f263057a5cbcaa
    (cherry picked from commit 10a523a03d6c6883015e620de52c64b85d20a188)
    (cherry picked from commit 3c71c99d7c843ac87616621a4eb084b0852b65f8)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 12.0.0

This issue was fixed in the openstack/tripleo-heat-templates 12.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 10.6.2

This issue was fixed in the openstack/tripleo-heat-templates 10.6.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 11.3.1

This issue was fixed in the openstack/tripleo-heat-templates 11.3.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.