rsync: readlink_stat(\"/etc/puppetlabs/facter/facter.conf\") failed: Permission denied (13)

Bug #1837082 reported by James Slagle
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Alex Schultz

Bug Description

Getting this error during "TASK [Debug output for task: Run container-puppet tasks (generate config) during step 1]":

        "2019-07-18 15:48:40,923 ERROR: 42805 -- ['/usr/bin/docker', 'run', '--user', 'root', '--name', u'container-puppet-crond', '--env', 'PUPPET_TAGS=file,file_line,concat,augeas,cron', '--env', u'NAME=crond', '--env', u'HO
STNAME=compute-1', '--env', 'NO_ARCHIVE=', '--env', 'STEP=6', '--env', 'NET_HOST=true', '--env', 'DEBUG=True', '--volume', '/etc/localtime:/etc/localtime:ro', '--volume', '/tmp/tmp6i3PhY:/etc/config.pp:ro', '--volume', '/etc/p
uppet/:/tmp/puppet-etc/:ro', '--volume', '/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume', '/etc/pki/tls/certs/ca-bund
le.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume', '/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume', '/var/lib/config-data:/var/lib/config-data/:rw', '--volume', '/var/lib/container-puppet/puppetl
abs/facter.conf:/etc/puppetlabs/facter/facter.conf:ro', '--volume', '/var/lib/container-puppet/puppetlabs/:/opt/puppetlabs/:ro', '--volume', '/dev/log:/dev/log:rw', '--volume', '/usr/share/openstack-puppet/modules/:/usr/share/
openstack-puppet/modules/:ro', '--entrypoint', '/var/lib/container-puppet/container-puppet.sh', '--net', 'host', '--volume', '/etc/hosts:/etc/hosts:ro', '--volume', '/var/lib/container-puppet/container-puppet.sh:/var/lib/conta
iner-puppet/container-puppet.sh:ro', u'docker.io/tripleomaster/centos-binary-cron:current-tripleo'] run failed after + mkdir -p /etc/puppet",
        "+ cp -dR /tmp/puppet-etc/auth.conf /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/hieradata /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet",
        "+ rm -Rf /etc/puppet/ssl",
        "+ echo '{\"step\": 6}'",
        "+ TAGS=",
        "+ '[' -n file,file_line,concat,augeas,cron ']'",
        "+ TAGS='--tags file,file_line,concat,augeas,cron'",
        "+ CHECK_MODE=",
        "+ '[' -d /tmp/puppet-check-mode ']'",
        "+ origin_of_time=/var/lib/config-data/crond.origin_of_time",
        "+ touch /var/lib/config-data/crond.origin_of_time",
        "+ sync",
        "+ export NET_HOST=true",
        "+ NET_HOST=true",
        "+ set +e",
        "+ '[' true == false ']'",
        "+ export FACTER_deployment_type=containers",
        "+ FACTER_deployment_type=containers",
        "++ cat /sys/class/dmi/id/product_uuid",
        "++ tr '[:upper:]' '[:lower:]'",
        "+ export FACTER_uuid=088060d1-af30-4434-9dd3-feb0253f4bcc",
        "+ FACTER_uuid=088060d1-af30-4434-9dd3-feb0253f4bcc",
        "+ echo 'Running puppet'",
        "+ set -x",
        "+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,cron /etc/config.pp",
        "+ logger -s -t puppet-user",
        "puppet-user: Warning: Support for ruby version 2.0.0 is deprecated and will be removed in a future release. See https://puppet.com/docs/puppet/latest/system_requirements.html for a list of supported ruby versions.",
        "puppet-user: (location: /usr/share/ruby/vendor_ruby/puppet.rb:130:in `<module:Puppet>')",
        "puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5",
        "puppet-user: (file: /etc/puppet/hiera.yaml)",
        "puppet-user: Warning: Undefined variable '::deploy_config_name'; \\n (file & line not available)",
        "puppet-user: Warning: ModuleLoader: module 'tripleo' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not avail
able)",
        "puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/5.5/deprecated_language.html\\n (file & line not available)",
        "puppet-user: Notice: Compiled catalog for compute-1.redhat.local in environment production in 1.39 seconds",
        "puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Logging::Logrotate/File[/etc/logrotate-crond.conf]/ensure: defined content as '{md5}4566116970cb9329655a0e08e1274dd0'",
        "puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Logging::Logrotate/Cron[logrotate-crond]/ensure: created",
        "puppet-user: Notice: Applied catalog in 0.41 seconds",
        "puppet-user: Changes:",
        "puppet-user: Total: 2",
        "puppet-user: Events:",
        "puppet-user: Success: 2",
        "puppet-user: Resources:",
        "puppet-user: Changed: 2",
        "puppet-user: Out of sync: 2",
        "puppet-user: Skipped: 7",
        "puppet-user: Total: 9",
        "puppet-user: Time:",
        "puppet-user: File: 0.06",
        "puppet-user: Cron: 0.09",
        "puppet-user: Transaction evaluation: 0.31",
        "puppet-user: Catalog application: 0.41",
        "puppet-user: Last run: 1563464914",
        "puppet-user: Config retrieval: 2.69",
        "puppet-user: Total: 0.42",
        "puppet-user: Version:",
        "puppet-user: Config: 1563464911",
        "puppet-user: Puppet: 5.5.10",
        "+ rc=2",
        "+ '[' True = false ']'",
        "+ set -e",
        "+ '[' 2 -ne 2 -a 2 -ne 0 ']'",
        "+ verbosity=",
        "+ verbosity=-v",
        "+ '[' -z '' ']'",
        "+ archivedirs=(\"/etc\" \"/root\" \"/opt\" \"/var/lib/ironic/tftpboot\" \"/var/lib/ironic/httpboot\" \"/var/www\" \"/var/spool/cron\" \"/var/lib/nova/.ssh\")",
        "+ rsync_srcs=",
        "+ for d in '\"${archivedirs[@]}\"'",
        "+ '[' -d /etc ']'",
        "+ rsync_srcs+=' /etc'",
        "+ '[' -d /root ']'",
        "+ rsync_srcs+=' /root'",
        "+ '[' -d /opt ']'",
        "+ rsync_srcs+=' /opt'",
        "+ '[' -d /var/lib/ironic/tftpboot ']'",
        "+ '[' -d /var/lib/ironic/httpboot ']'",
        "+ '[' -d /var/www ']'",
        "+ '[' -d /var/spool/cron ']'",
        "+ rsync_srcs+=' /var/spool/cron'",
        "+ '[' -d /var/lib/nova/.ssh ']'",
        "+ password_files=/root/.my.cnf",
        "+ exclude_files=",
        "+ for p in '$password_files'",
        "+ '[' -f /root/.my.cnf -a -f /var/lib/config-data/crond/root/.my.cnf ']'",
        "+ echo 'Rsyncing config files from /etc /root /opt /var/spool/cron into /var/lib/config-data/crond'",
        "+ rsync -a -v -R --delay-updates --delete-after /etc /root /opt /var/spool/cron /var/lib/config-data/crond",
        "rsync: readlink_stat(\"/etc/puppetlabs/facter/facter.conf\") failed: Permission denied (13)",
        "rsync: readlink_stat(\"/opt/puppetlabs/facter.conf\") failed: Permission denied (13)",
        "rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]",
        " attempt(s): 1",

I'm using latest repos from tripleo-repos current-tripleo-dev as of July 18, 2019.

Revision history for this message
James Slagle (james-slagle) wrote :

I'm using pre-provisioned nodes, and the nodes are fully up to date with the latest repos from "tripleo-repos current-tripleo-dev"

[root@compute-1 yum.repos.d]# cat delorean.repo
[delorean]
name=delorean-ansible-config_template-dc899ca92eb227ade3c95e5511a42814f67a9d11
baseurl=https://trunk.rdoproject.org/centos7/dc/89/dc899ca92eb227ade3c95e5511a42814f67a9d11_e2fc0ef1
enabled=1
gpgcheck=0
priority=10
includepkgs=instack,instack-undercloud,os-apply-config,os-collect-config,os-net-config,os-refresh-config,python*-tripleoclient,openstack-tripleo-*,openstack-puppet-modules,puppet-*,python*-tripleo-common,python*-paunch*[root@compute-1 yum.repos.d]# cat delorean-current-tripleo.repo
[delorean-current-tripleo]
name=delorean-tripleo-ansible-69cab4cd5356e6c5314a103ca760d531d004dc5a
baseurl=https://trunk.rdoproject.org/centos7/69/ca/69cab4cd5356e6c5314a103ca760d531d004dc5a_802178e9
enabled=1
gpgcheck=0
priority=20[root@compute-1 yum.repos.d]#

Changed in tripleo:
importance: Undecided → High
milestone: none → train-2
status: New → In Progress
assignee: nobody → Alex Schultz (alex-schultz)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/671524

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/671522
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1565c50be6c5d7012eec74a9731efd93c4d77119
Submitter: Zuul
Branch: master

commit 1565c50be6c5d7012eec74a9731efd93c4d77119
Author: Alex Schultz <email address hidden>
Date: Thu Jul 18 16:29:44 2019 +0000

    Revert "Add facter cache for container configurations"

    This reverts commit 4ec57c60f30f994d8bf36021ff12670e6c8c77b5.

    There are some perm issues with the config generation. Let's
    revert this and address those.

    Related-Bug: #1837082
    Change-Id: Ia806fba30724f18a21153557472e9fb493939ced

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/671524
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=08e34dbcaebb0608df61d6540d4919cab71632f5
Submitter: Zuul
Branch: master

commit 08e34dbcaebb0608df61d6540d4919cab71632f5
Author: Alex Schultz <email address hidden>
Date: Thu Jul 18 10:36:05 2019 -0600

    Re-Add facter cache for container configurations

    Rather than running all the facts for every invocation of
    container-puppet.py, let's cache the facts for the entire container
    config step. This is similar to what used to happen when we ran puppet
    in a single instance for a given step. This improves the time it takes
    puppet tp execute within the containers. When systems have large number
    of interfaces, the fact generation for puppet can come to a crawl given
    that puppet is not supposed to be invoked multiple times in parallel. By
    pre-caching the facts, we eliminate the need for puppet to do the
    network (and other fact) related lookups when running container
    configuration generation.

    This patch includes logic to exclude the puppet caching dirs that we
    mount as read only in the config containers.

    Change-Id: I94002572ec7ac1767fbfbfdf4c6fa6fbf0cd58d5
    Closes-Bug: #1837082
    Closes-Bug: #1835959

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/672047

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/672048

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/672049

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 11.1.0

This issue was fixed in the openstack/tripleo-heat-templates 11.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.opendev.org/672049
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ef01df9b8df79f7faf21791e9d9ddcd85e706bd4
Submitter: Zuul
Branch: stable/queens

commit ef01df9b8df79f7faf21791e9d9ddcd85e706bd4
Author: Alex Schultz <email address hidden>
Date: Thu Jul 18 10:36:05 2019 -0600

    Re-Add facter cache for container configurations

    Rather than running all the facts for every invocation of
    container-puppet.py, let's cache the facts for the entire container
    config step. This is similar to what used to happen when we ran puppet
    in a single instance for a given step. This improves the time it takes
    puppet tp execute within the containers. When systems have large number
    of interfaces, the fact generation for puppet can come to a crawl given
    that puppet is not supposed to be invoked multiple times in parallel. By
    pre-caching the facts, we eliminate the need for puppet to do the
    network (and other fact) related lookups when running container
    configuration generation.

    This patch includes logic to exclude the puppet caching dirs that we
    mount as read only in the config containers.

    Conflicts:
     common/container-puppet.py

    Change-Id: I94002572ec7ac1767fbfbfdf4c6fa6fbf0cd58d5
    Closes-Bug: #1837082
    Closes-Bug: #1835959
    (cherry picked from commit 08e34dbcaebb0608df61d6540d4919cab71632f5)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.opendev.org/672048
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c6a76f9263c52320e24f000ca0c46b643f2d2852
Submitter: Zuul
Branch: stable/rocky

commit c6a76f9263c52320e24f000ca0c46b643f2d2852
Author: Alex Schultz <email address hidden>
Date: Thu Jul 18 10:36:05 2019 -0600

    Re-Add facter cache for container configurations

    Rather than running all the facts for every invocation of
    container-puppet.py, let's cache the facts for the entire container
    config step. This is similar to what used to happen when we ran puppet
    in a single instance for a given step. This improves the time it takes
    puppet tp execute within the containers. When systems have large number
    of interfaces, the fact generation for puppet can come to a crawl given
    that puppet is not supposed to be invoked multiple times in parallel. By
    pre-caching the facts, we eliminate the need for puppet to do the
    network (and other fact) related lookups when running container
    configuration generation.

    This patch includes logic to exclude the puppet caching dirs that we
    mount as read only in the config containers.

    Conflicts:
     common/container-puppet.py

    Change-Id: I94002572ec7ac1767fbfbfdf4c6fa6fbf0cd58d5
    Closes-Bug: #1837082
    Closes-Bug: #1835959
    (cherry picked from commit 08e34dbcaebb0608df61d6540d4919cab71632f5)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/672047
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=10e8506e165a8aa8be7512dcd095bc661f7a347b
Submitter: Zuul
Branch: stable/stein

commit 10e8506e165a8aa8be7512dcd095bc661f7a347b
Author: Alex Schultz <email address hidden>
Date: Thu Jul 18 10:36:05 2019 -0600

    Re-Add facter cache for container configurations

    Rather than running all the facts for every invocation of
    container-puppet.py, let's cache the facts for the entire container
    config step. This is similar to what used to happen when we ran puppet
    in a single instance for a given step. This improves the time it takes
    puppet tp execute within the containers. When systems have large number
    of interfaces, the fact generation for puppet can come to a crawl given
    that puppet is not supposed to be invoked multiple times in parallel. By
    pre-caching the facts, we eliminate the need for puppet to do the
    network (and other fact) related lookups when running container
    configuration generation.

    This patch includes logic to exclude the puppet caching dirs that we
    mount as read only in the config containers.

    Conflicts:
     common/container-puppet.py

    Change-Id: I94002572ec7ac1767fbfbfdf4c6fa6fbf0cd58d5
    Closes-Bug: #1837082
    Closes-Bug: #1835959
    (cherry picked from commit 08e34dbcaebb0608df61d6540d4919cab71632f5)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 10.6.1

This issue was fixed in the openstack/tripleo-heat-templates 10.6.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 9.4.1

This issue was fixed in the openstack/tripleo-heat-templates 9.4.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 8.4.1

This issue was fixed in the openstack/tripleo-heat-templates 8.4.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.