swift-storage config-changed hook runs chown on /srv/node

Bug #1676728 reported by Brad Marshall
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Swift Storage Charm
Fix Released
High
Billy Olsen

Bug Description

On the current swift-storage charm (checked on both released 17.02 and master), it calls the setup_storage() function, which does a chown on /srv/node. On a busy swift system, this can be multiple terabytes and as you can imagine, this takes a very long time to run. As there's a hook running, this blocks any further activity on that unit, as well as the ability to use juju run.

Are there any other options for this rather than running this very expensive operation on every config changed hook run? Is just leaving it in the install hook sufficient?

Revision history for this message
Jill Rouleau (jillrouleau) wrote :

As an example, on a cloud with 5TB used storage (millions of objects) per node this operation had been running since March 31 without completion.

Changed in charm-swift-storage:
status: New → Triaged
importance: Undecided → High
milestone: none → 17.05
Revision history for this message
Drew Freiberger (afreiberger) wrote :

I definitely think there should either be a charm-local setting that tracks if the user or group changed in the charm config and run the chown in the config-changed if that's what was updated. Or, better yet, provide a config like 'chown-on-config-change=false' and override if user/group is in config keys changed.

Revision history for this message
Drew Freiberger (afreiberger) wrote :

I have found that the config-changed calls setup_storage() from lib/swift_storage_utils.py without any "smarts" in either to check if the chown/chmod is needed. Even doing a recursive getfacl would be 1000x faster than forcing 2 ACL write updates to determine if chown/chmod are necessary.

James Page (james-page)
Changed in charm-swift-storage:
milestone: 17.05 → 17.08
Revision history for this message
Billy Olsen (billy-olsen) wrote :

Patch proposed to master for the bug - https://review.openstack.org/#/c/472471/

Changed in charm-swift-storage:
assignee: nobody → Billy Olsen (billy-olsen)
status: Triaged → In Progress
tags: added: backport-potential sts
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-swift-storage (master)

Reviewed: https://review.openstack.org/472471
Committed: https://git.openstack.org/cgit/openstack/charm-swift-storage/commit/?id=d6061caa2cb1ab13ffd476a6d7f39b5b7ff75803
Submitter: Jenkins
Branch: master

commit d6061caa2cb1ab13ffd476a6d7f39b5b7ff75803
Author: Billy Olsen <email address hidden>
Date: Thu Jun 8 16:01:17 2017 -0700

    Only change owner/permissions of new devices

    Do not change owner and permissions of already existing
    devices in the setup_storage() function as this runs
    during every config-changed hook invocation.

    Change-Id: I21f23aee34d315ccb4df303527b4d791fc043f58
    Closes-Bug: #1676728

Changed in charm-swift-storage:
status: In Progress → Fix Committed
James Page (james-page)
Changed in charm-swift-storage:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.