swift: container swift_rsyncd can take hours to start because of chown on /srv/node

[Martin Chlumsky]

On master branch right now (and at least stein as well), the container swift_rsyncd runs the following command when starting (see extend_start.sh) before starting rsyncd:

sudo chown -R swift:swift /srv/node

This can take very long to run (hours) depending on the how much data there is already on disk.

To make matters worse, some of the other swift containers delete files while chown is running which causes it to exit with a non-zero status code and the container start fails and restarts the chown again.

I think it should be up to the operator to do the chown when preparing the disks or at the very least make running the chown configurable.

[Mark Goddard (mgoddard)]

Thanks for raising this. We have seen other similar issues, where we're a little overzealous with recursive permissions.

Is this command actually necessary? Files should be created by the swift services which run as the swift user, right? Perhaps we just need to chown the mount point, /srv/node?

[Martin Chlumsky (martin-chlumsky)]

I think /srv/node needs to be chown'ed and so do the mountpoints under /srv/node (just not recursively).

The documentation[1] doesn't specify that any chowning needs to be done at all so it should be done by kolla-ansible.
I would add the chowns in the start task [2].

[1] https://docs.openstack.org/kolla-ansible/latest/reference/storage/swift-guide.html#disks-with-a-partition-table-recommended
[2] https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/swift/tasks/start.yml#L35

[Mark Goddard (mgoddard)]

I don't think we can do it in the start task - it needs to be done in the containers. We can either keep this in extend_start.sh, or add it to the config.json file for each container.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/692870

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692870
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=2d122afaf9817cadc668b308c4f736e47c09a55f
Submitter: Zuul
Branch: master

commit 2d122afaf9817cadc668b308c4f736e47c09a55f
Author: Martin Chlumsky <email address hidden>
Date: Mon Nov 4 13:46:49 2019 -0500

    swift-rsyncd: don't recursively chown /srv/node at startup

    Instead of doing a recursive chown on /srv/node, chown only /srv/node
    and any immediate subdirectory under /srv/node.

    Change-Id: I8fd93fa77ba9fc63910360300b8783a34bd4e6d5
    Closes-Bug: #1851169

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/693575

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/693576

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/693668

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693575
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=78762fabbd7440d699b6cb52c1da94a7529eb39c
Submitter: Zuul
Branch: stable/train

commit 78762fabbd7440d699b6cb52c1da94a7529eb39c
Author: Martin Chlumsky <email address hidden>
Date: Mon Nov 4 13:46:49 2019 -0500

    swift-rsyncd: don't recursively chown /srv/node at startup

    Instead of doing a recursive chown on /srv/node, chown only /srv/node
    and any immediate subdirectory under /srv/node.

    Change-Id: I8fd93fa77ba9fc63910360300b8783a34bd4e6d5
    Closes-Bug: #1851169
    (cherry picked from commit 2d122afaf9817cadc668b308c4f736e47c09a55f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693576
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=8673f02844a4a43e6eeb74d6437c2d97b530802a
Submitter: Zuul
Branch: stable/stein

commit 8673f02844a4a43e6eeb74d6437c2d97b530802a
Author: Martin Chlumsky <email address hidden>
Date: Mon Nov 4 13:46:49 2019 -0500

    swift-rsyncd: don't recursively chown /srv/node at startup

    Instead of doing a recursive chown on /srv/node, chown only /srv/node
    and any immediate subdirectory under /srv/node.

    Change-Id: I8fd93fa77ba9fc63910360300b8783a34bd4e6d5
    Closes-Bug: #1851169
    (cherry picked from commit 2d122afaf9817cadc668b308c4f736e47c09a55f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693668
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=daafc9380b326cb04ced3033dd8686673df3dc0d
Submitter: Zuul
Branch: stable/rocky

commit daafc9380b326cb04ced3033dd8686673df3dc0d
Author: Martin Chlumsky <email address hidden>
Date: Mon Nov 4 13:46:49 2019 -0500

    swift-rsyncd: don't recursively chown /srv/node at startup

    Instead of doing a recursive chown on /srv/node, chown only /srv/node
    and any immediate subdirectory under /srv/node.

    Change-Id: I8fd93fa77ba9fc63910360300b8783a34bd4e6d5
    Closes-Bug: #1851169
    (cherry picked from commit 2d122afaf9817cadc668b308c4f736e47c09a55f)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 9.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 7.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 8.0.2 release.