Need adjustable partition power for swift

Bug #1716766 reported by Drew Freiberger
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Swift Proxy Charm
Confirmed
Medium
Unassigned

Bug Description

We have an environment that was set with the default partition power (8) which caused there to be 256 partitions across 3 zones with 3 hosts each, but the usage on this environment is over 500,000 objects. The partition power on this environment needs to be raised from 8 to 18, but it appears that this ability is not available in the product at the moment.

This bug is to track a request to progress and complete implementation of the following spec:

https://specs.openstack.org/openstack/swift-specs/specs/in_progress/increasing_partition_power.html

The current partition power causes over 12 hours of replication lag which then puts us into contention with another issue noted in lp:1691570 with hanging and zombie rsync processes due to "lockup detected killing coros" not actually cleaning up coros fully.

Using Mitaka on Trusty with Swift 2.7.1 and charm version 17.02.

You can likely replicate the issue in lp:1691570 with a swift environment setup on xfs backed storage with over 2.5 million objects per partition.

Revision history for this message
Edward Hope-Morley (hopem) wrote :

Hi Drew, it should be possible to modify partition power with the charms today. There are some conditions that need to be met in order for it to work though. To help me figure out what's going on can you please provide the following:

  * steps (charm config changes etc) you tried to change partition power

  * your swift-proxy charm config

  * /var/log/juju/unit-swift-proxy-<id>.log from your leader unit (dump in a pastebin if possible)

  * output of 'swift-ring-builder /etc/swift/object.ring.gz'

Changed in charm-swift-proxy:
status: New → Incomplete
Revision history for this message
Edward Hope-Morley (hopem) wrote :

Ok I've had a look at the swift-proxy charm code to refresh my memory on how this is currently working and can confirm that partition power is only used by the charm the builders are first created so modifying this config option post-deployment will not have any effect. This is in fact the intended behaviour i.e. we just didn't add support for this to be changed. I assume that right now, users of this charm are manually updated the builders and re-spinning and syncing the rings manually. As previously mentioned this is not advisable since it will eventually conflict with the charms handling ring maintenance the next time it tries to update the rings in any way. So in my view the way forward is to extend what we have to be able to support updating this setting post-deployment while ensuring that we are able to do this without putting the cluster at-risk. I can see how this might seem suitable as a charm action but since the code that currently handles rings and builders is not controlled by actions (it actually pre-dates actions) and relies on charm config that actions can't set, it probably going to make most sense to extend the existing hook code. I intend to work on a patch to do this and document how it should be done so will post back once i have something.

Changed in charm-swift-proxy:
status: Incomplete → New
status: New → Confirmed
milestone: none → 17.11
importance: Undecided → Medium
James Page (james-page)
Changed in charm-swift-proxy:
milestone: 17.11 → 18.02
Ryan Beisner (1chb1n)
Changed in charm-swift-proxy:
milestone: 18.02 → 18.05
David Ames (thedac)
Changed in charm-swift-proxy:
milestone: 18.05 → 18.08
James Page (james-page)
Changed in charm-swift-proxy:
milestone: 18.08 → 18.11
David Ames (thedac)
Changed in charm-swift-proxy:
milestone: 18.11 → 19.04
David Ames (thedac)
Changed in charm-swift-proxy:
milestone: 19.04 → 19.07
David Ames (thedac)
Changed in charm-swift-proxy:
milestone: 19.07 → 19.10
David Ames (thedac)
Changed in charm-swift-proxy:
milestone: 19.10 → 20.01
James Page (james-page)
Changed in charm-swift-proxy:
milestone: 20.01 → 20.05
David Ames (thedac)
Changed in charm-swift-proxy:
milestone: 20.05 → 20.08
James Page (james-page)
Changed in charm-swift-proxy:
milestone: 20.08 → none
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.