tripleo

containers stuck in restarting. Error is OSError: [Errno 30] Read-only file system: '/etc/pki/ca-trust/extracted'

Bug #1860607 reported by Alex Schultz
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Alex Schultz
tripleo ussuri-2 "tripleo ussuri-2"

Bug Description

In some case there may be a race condition in the container-puppet.sh (or docker-puppet.sh) where the read-only mounts end up getting copied into /var/lib/config-data/puppet-generated/<container>/. When the file ends up here, kolla will try to copy the data over from this folder into a ro mount. This usually happens to /etc/pki/ca-trust/extracted where this folder is updated while a container puppet execution is being run. This results in the affected containers failing with the following on startup:

Jan 20 13:26:38 compute13 journal: INFO:__main__:Deleting /etc/neutron/neutron.conf
Jan 20 13:26:38 compute13 journal: INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/neutron/neutron.conf to /etc/neutron/neutron.conf
Jan 20 13:26:38 compute13 journal: ERROR:__main__:Unexpected error:
Jan 20 13:26:38 compute13 journal: Traceback (most recent call last):
Jan 20 13:26:38 compute13 journal: File "/usr/local/bin/kolla_set_configs", line 411, in main
Jan 20 13:26:38 compute13 journal: execute_config_strategy(config)
Jan 20 13:26:38 compute13 journal: File "/usr/local/bin/kolla_set_configs", line 377, in execute_config_strategy
Jan 20 13:26:38 compute13 journal: copy_config(config)
Jan 20 13:26:38 compute13 journal: File "/usr/local/bin/kolla_set_configs", line 306, in copy_config
Jan 20 13:26:38 compute13 journal: config_file.copy()
Jan 20 13:26:38 compute13 journal: File "/usr/local/bin/kolla_set_configs", line 150, in copy
Jan 20 13:26:38 compute13 journal: self._merge_directories(source, dest)
Jan 20 13:26:38 compute13 journal: File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories
Jan 20 13:26:38 compute13 journal: os.path.join(dest, to_copy))
Jan 20 13:26:38 compute13 journal: File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories
Jan 20 13:26:38 compute13 journal: os.path.join(dest, to_copy))
Jan 20 13:26:38 compute13 journal: File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories
Jan 20 13:26:38 compute13 journal: os.path.join(dest, to_copy))
Jan 20 13:26:38 compute13 journal: File "/usr/local/bin/kolla_set_configs", line 92, in _merge_directories
Jan 20 13:26:38 compute13 journal: self._set_properties(source, dest)
Jan 20 13:26:38 compute13 journal: File "/usr/local/bin/kolla_set_configs", line 117, in _set_properties
Jan 20 13:26:38 compute13 journal: self._set_properties_from_file(source, dest)
Jan 20 13:26:38 compute13 journal: File "/usr/local/bin/kolla_set_configs", line 122, in _set_properties_from_file
Jan 20 13:26:38 compute13 journal: shutil.copystat(source, dest)
Jan 20 13:26:38 compute13 journal: File "/usr/lib64/python2.7/shutil.py", line 98, in copystat
Jan 20 13:26:38 compute13 journal: os.utime(dst, (st.st_atime, st.st_mtime))
Jan 20 13:26:38 compute13 journal: OSError: [Errno 30] Read-only file system: '/etc/pki/ca-trust/extracted'

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/703873

Changed in tripleo:
assignee: nobody → Alex Schultz (alex-schultz)
status: Triaged → In Progress
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

> This usually happens to /etc/pki/ca-trust/extracted where this folder is updated while a container puppet execution is being run.

I wonder if using the overlay-type mount could help to ingore any changes in the "upper" and only kepp it going on with the "lower" contents?

OpenStack Infra (hudson-openstack)
Changed in tripleo:
assignee: Alex Schultz (alex-schultz) → Bogdan Dobrelya (bogdando)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/703873
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=36ea0148e8b2ed30fb0a03fe6f812fd00657924b
Submitter: Zuul
Branch: master

commit 36ea0148e8b2ed30fb0a03fe6f812fd00657924b
Author: Alex Schultz <email address hidden>
Date: Wed Jan 22 14:24:41 2020 -0700

    Update ro excludes

    When container-puppet.sh (or docker-puppet.sh) runs, we always bind
    mount in certain directories and files. However if these files change
    while we're running the puppet execution externally on the host, they
    end up being copied to our puppet-generated path. This path is copied
    into the container when kolla starts up. Because we re-mount these same
    paths in the end container as read-only, kolla will try and update the
    read-only mount which errors. This change updates our rsync excludes to
    include:

     * /etc/pki/ca-trust/extracted
     * /etc/pki/ca-trust/source/anchors
     * /etc/pki/tls/certs/ca-bundle.crt
     * /etc/pki/tls/certs/ca-bundle.trust.crt
     * /etc/pki/tls/cert.pem
     * /etc/hosts
     * /etc/localtime

    Change-Id: I858a7b8745688c6271b01795460ab4adbf3f2be5
    Closes-Bug: #1860607

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/706521

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/706530

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/706531

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/706532

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/706530
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=bbfb02b2d0929dccfed67014d3cc06a1ffaf2a1c
Submitter: Zuul
Branch: stable/stein

commit bbfb02b2d0929dccfed67014d3cc06a1ffaf2a1c
Author: Alex Schultz <email address hidden>
Date: Wed Jan 22 14:24:41 2020 -0700

    Update ro excludes

    When container-puppet.sh (or docker-puppet.sh) runs, we always bind
    mount in certain directories and files. However if these files change
    while we're running the puppet execution externally on the host, they
    end up being copied to our puppet-generated path. This path is copied
    into the container when kolla starts up. Because we re-mount these same
    paths in the end container as read-only, kolla will try and update the
    read-only mount which errors. This change updates our rsync excludes to
    include:

     * /etc/pki/ca-trust/extracted
     * /etc/pki/ca-trust/source/anchors
     * /etc/pki/tls/certs/ca-bundle.crt
     * /etc/pki/tls/certs/ca-bundle.trust.crt
     * /etc/pki/tls/cert.pem
     * /etc/hosts
     * /etc/localtime

    Conflicts:
            common/container-puppet.sh

    Change-Id: I858a7b8745688c6271b01795460ab4adbf3f2be5
    Closes-Bug: #1860607
    (cherry picked from commit 36ea0148e8b2ed30fb0a03fe6f812fd00657924b)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 12.1.0

This issue was fixed in the openstack/tripleo-heat-templates 12.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/train)

Reviewed: https://review.opendev.org/706521
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c8144799e8ef125437f74ef4911923a487970854
Submitter: Zuul
Branch: stable/train

commit c8144799e8ef125437f74ef4911923a487970854
Author: Alex Schultz <email address hidden>
Date: Wed Jan 22 14:24:41 2020 -0700

    Update ro excludes

    When container-puppet.sh (or docker-puppet.sh) runs, we always bind
    mount in certain directories and files. However if these files change
    while we're running the puppet execution externally on the host, they
    end up being copied to our puppet-generated path. This path is copied
    into the container when kolla starts up. Because we re-mount these same
    paths in the end container as read-only, kolla will try and update the
    read-only mount which errors. This change updates our rsync excludes to
    include:

     * /etc/pki/ca-trust/extracted
     * /etc/pki/ca-trust/source/anchors
     * /etc/pki/tls/certs/ca-bundle.crt
     * /etc/pki/tls/certs/ca-bundle.trust.crt
     * /etc/pki/tls/cert.pem
     * /etc/hosts
     * /etc/localtime

    Change-Id: I858a7b8745688c6271b01795460ab4adbf3f2be5
    Closes-Bug: #1860607
    (cherry picked from commit 36ea0148e8b2ed30fb0a03fe6f812fd00657924b)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.opendev.org/706531
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=e0485ea44f72ff39e96947bdfca6846b1ec4f2a1
Submitter: Zuul
Branch: stable/rocky

commit e0485ea44f72ff39e96947bdfca6846b1ec4f2a1
Author: Alex Schultz <email address hidden>
Date: Wed Jan 22 14:24:41 2020 -0700

    Update ro excludes

    When container-puppet.sh (or docker-puppet.sh) runs, we always bind
    mount in certain directories and files. However if these files change
    while we're running the puppet execution externally on the host, they
    end up being copied to our puppet-generated path. This path is copied
    into the container when kolla starts up. Because we re-mount these same
    paths in the end container as read-only, kolla will try and update the
    read-only mount which errors. This change updates our rsync excludes to
    include:

     * /etc/pki/ca-trust/extracted
     * /etc/pki/ca-trust/source/anchors
     * /etc/pki/tls/certs/ca-bundle.crt
     * /etc/pki/tls/certs/ca-bundle.trust.crt
     * /etc/pki/tls/cert.pem
     * /etc/hosts
     * /etc/localtime

    Conflicts:
            common/container-puppet.sh

    Change-Id: I858a7b8745688c6271b01795460ab4adbf3f2be5
    Closes-Bug: #1860607
    (cherry picked from commit 36ea0148e8b2ed30fb0a03fe6f812fd00657924b)

tags: added: in-stable-rocky
tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.opendev.org/706532
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1f0d31e5d5ea5f4423ee12df56c42ef6ddf8a7b5
Submitter: Zuul
Branch: stable/queens

commit 1f0d31e5d5ea5f4423ee12df56c42ef6ddf8a7b5
Author: Alex Schultz <email address hidden>
Date: Wed Jan 22 14:24:41 2020 -0700

    Update ro excludes

    When container-puppet.sh (or docker-puppet.sh) runs, we always bind
    mount in certain directories and files. However if these files change
    while we're running the puppet execution externally on the host, they
    end up being copied to our puppet-generated path. This path is copied
    into the container when kolla starts up. Because we re-mount these same
    paths in the end container as read-only, kolla will try and update the
    read-only mount which errors. This change updates our rsync excludes to
    include:

     * /etc/pki/ca-trust/extracted
     * /etc/pki/ca-trust/source/anchors
     * /etc/pki/tls/certs/ca-bundle.crt
     * /etc/pki/tls/certs/ca-bundle.trust.crt
     * /etc/pki/tls/cert.pem
     * /etc/hosts
     * /etc/localtime

    Conflicts:
            common/container-puppet.sh

    Change-Id: I858a7b8745688c6271b01795460ab4adbf3f2be5
    Closes-Bug: #1860607
    (cherry picked from commit 36ea0148e8b2ed30fb0a03fe6f812fd00657924b)

Bogdan Dobrelya (bogdando)
Changed in tripleo:
assignee: Bogdan Dobrelya (bogdando) → Alex Schultz (alex-schultz)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 11.4.0

This issue was fixed in the openstack/tripleo-heat-templates 11.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates rocky-eol

This issue was fixed in the openstack/tripleo-heat-templates rocky-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates queens-eol

This issue was fixed in the openstack/tripleo-heat-templates queens-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates stein-eol

This issue was fixed in the openstack/tripleo-heat-templates stein-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.