scenario 1 & 2 standalone timeout "Evaluation Error: ... Class[Redis]: parameter 'ulimit' expects an Integer value"

Bug #1854136 reported by Marios Andreou
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Michele Baldessari

Bug Description

at [1][2] the tripleo-ci-centos-7-scenario002-standalone and scenario001-standalone are timing out with an error during standalone deploy - trace like:

        * 2019-11-26 18:00:16 | "2019-11-26 17:58:33,430 ERROR: 86591 -- ['/usr/bin/podman', 'run', '--user', 'root', '--name', u'container-puppet-redis', '--env', u'PUPPET_TAGS=file,file_line,concat,augeas,cron,exec', '--env', u'NAME=redis', '--env', 'HOSTNAME=standalone', '--env', 'NO_ARCHIVE=', '--env', 'STEP=6', '--env', 'NET_HOST=true', '--env', 'DEBUG=False', '--volume', '/etc/localtime:/etc/localtime:ro', '--volume', '/tmp/tmpgZBgil:/etc/config.pp:ro', '--volume', '/etc/puppet/:/tmp/puppet-etc/:ro', '--volume', '/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume', '/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume', '/var/lib/config-data:/var/lib/config-data/:rw', '--volume', '/var/lib/container-puppet/puppetlabs/facter.conf:/etc/puppetlabs/facter/facter.conf:ro', '--volume', '/var/lib/container-puppet/puppetlabs/:/opt/puppetlabs/:ro', '--volume', '/dev/log:/dev/log:rw', '--rm', '--log-driver', 'k8s-file', '--log-opt', u'path=/var/log/containers/stdouts/container-puppet-redis.log', '--security-opt', 'label=disable', '--volume', '/usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro', '--entrypoint', '/var/lib/container-puppet/container-puppet.sh', '--net', 'host', '--volume', '/etc/hosts:/etc/hosts:ro', '--volume', '/var/lib/container-puppet/container-puppet.sh:/var/lib/container-puppet/container-puppet.sh:ro', u'192.168.24.1:8787/tripleomaster/centos-binary-redis:e49ccfbdc1b555a277087b9e497285a824911408_6682698d-updated-20191126171831'] run failed after + mkdir -p /etc/puppet",

        * 2019-11-26 18:00:16 | "puppet-user: Error: Evaluation Error: Error while evaluating a Resource Statement, Class[Redis]: parameter 'ulimit' expects an Integer value, got String (file: /etc/puppet/modules/tripleo/manifests/profile/base/database/redis.pp, line: 110, column: 5) on node standalone.localdomain",

blocks tripleo gates lots/most *tripleo* run these jobs

[1] https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b24/693355/13/gate/tripleo-ci-centos-7-scenario001-standalone/b24dc4d/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
[2] https://e3ba71bffa00fbfa476f-e0419d2da2313928ed85c22410bb6c0a.ssl.cf1.rackcdn.com/695575/6/gate/tripleo-ci-centos-7-scenario002-standalone/d07f2db/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

Revision history for this message
Michele Baldessari (michele) wrote :
Changed in tripleo:
assignee: nobody → Michele Baldessari (michele)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/696285

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
Marios Andreou (marios-b) wrote :

also now a promotion blocker seen in centos periodics http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario002-standalone-master/7a02be6/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

 2019-11-27 02:20:02 | "2019-11-27 02:18:54,064 ERROR: 59702 -- ['/usr/bin/podman', 'run', '--user', 'root', '--name', u'container-puppet-redis', '--env', u'PUPPET_TAGS=file,file_line,concat,augeas,cron,exec', '--env', u'NAME=redis', '--env', 'HOSTNAME=standalone', '--env', 'NO_ARCHIVE=', '--env', 'STEP=6', '--env', 'NET_HOST=true', '--env', 'DEBUG=True', '--volume', '/etc/localtime:/etc/localtime:ro', '--volume', '/tmp/tmp__pJAd:/etc/config.pp:ro', '--volume', '/etc/puppet/:/tmp/puppet-etc/:ro', '--volume', '/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume', '/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume', '/var/lib/config-data:/var/lib/config-data/:rw', '--volume', '/var/lib/container-puppet/puppetlabs/facter.conf:/etc/puppetlabs/facter/facter.conf:ro', '--volume', '/var/lib/container-puppet/puppetlabs/:/opt/puppetlabs/:ro', '--volume', '/dev/log:/dev/log:rw', '--rm', '--log-driver', 'k8s-file', '--log-opt', u'path=/var/log/containers/stdouts/container-puppet-redis.log', '--security-opt', 'label=disable', '--volume', '/usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro', '--entrypoint', '/var/lib/container-puppet/container-puppet.sh', '--net', 'host', '--volume', '/etc/hosts:/etc/hosts:ro', '--volume', '/var/lib/container-puppet/container-puppet.sh:/var/lib/container-puppet/container-puppet.sh:ro', u'192.168.24.1:8787/tripleomaster/centos-binary-redis:ce6b2e80e2d8aed52daa60d30494909aec3df391_cccbfe2b-updated-20191127015757'] run failed after + mkdir -p /etc/puppet",

tags: added: promotion-blocker
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/696285
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b82725f5e097e962bc9181fb18ee32a13c5ad649
Submitter: Zuul
Branch: master

commit b82725f5e097e962bc9181fb18ee32a13c5ad649
Author: Michele Baldessari <email address hidden>
Date: Wed Nov 27 12:16:34 2019 +0100

    Force redis::ulimit to be an integer

    The upstream change at:
    https://github.com/voxpupuli/puppet-redis/commit/502ff38c8c035fa143e32dd4ee61b3ec68db461c

    made redis::ulimit an integer and will now error out if passed a string.
    Historically in THT we passed RedisFDLimit directly to the redis::ulimit
    hiera key as a string. Let's just change RedisFDLimit to a number. The
    only affected people will be those that A) tweaked this parameter and
    B) actually wrapped the param into quotes so to force it as a string.
    Given that this number is likely non-significance, we choose this route
    and we avoid an extra YAQL invocation to do a conversion in THT.

    Change-Id: I70361801c71738f2d95930aea99914316d9f0d11
    Closes-Bug: #1854136

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/696417

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/train)

Reviewed: https://review.opendev.org/696417
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=8002eb7df22c98b06d7d212bf5748d3f9389880f
Submitter: Zuul
Branch: stable/train

commit 8002eb7df22c98b06d7d212bf5748d3f9389880f
Author: Michele Baldessari <email address hidden>
Date: Wed Nov 27 12:16:34 2019 +0100

    Force redis::ulimit to be an integer

    The upstream change at:
    https://github.com/voxpupuli/puppet-redis/commit/502ff38c8c035fa143e32dd4ee61b3ec68db461c

    made redis::ulimit an integer and will now error out if passed a string.
    Historically in THT we passed RedisFDLimit directly to the redis::ulimit
    hiera key as a string. Let's just change RedisFDLimit to a number. The
    only affected people will be those that A) tweaked this parameter and
    B) actually wrapped the param into quotes so to force it as a string.
    Given that this number is likely non-significance, we choose this route
    and we avoid an extra YAQL invocation to do a conversion in THT.

    Change-Id: I70361801c71738f2d95930aea99914316d9f0d11
    Closes-Bug: #1854136
    (cherry picked from commit b82725f5e097e962bc9181fb18ee32a13c5ad649)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 11.3.1

This issue was fixed in the openstack/tripleo-heat-templates 11.3.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 12.1.0

This issue was fixed in the openstack/tripleo-heat-templates 12.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.