swift-proxy showing split brain behaviour when deployed in HA mode
Bug #1337765 reported by
Alexander List
This bug affects 2 people
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| swift-proxy (Juju Charms Collection) |
Fix Released
|
High
|
JuanJo Ciarlante | ||
Bug Description
We are trying to deploy swift-proxy in HA mode, using 3 proxies in LXC containers on physical machines managed by MAAS (smoosh).
We see inconsistent answers from swift, pointing at a split-brain situation.
When reducing the number of proxies to one and removing HA, the problem is not visible.
We will do further testing on CI to try and reproduce the problem in a clean environment.
This is on trusty, using the latest version of the charm.
Related branches
lp:~jjo/charms/trusty/swift-proxy/swift_hash-from-JUJU_ENV_UUID-and-service_name
- Edward Hope-Morley: Approve
- James Page: Needs Information
-
Diff: 34 lines (+5/-5)1 file modifiedhooks/swift_context.py (+5/-5)
- OpenStack Charmers: Pending requested
-
Diff: 5652 lines (+4171/-249) (has conflicts)52 files modified.bzrignore (+2/-0)
Makefile (+24/-1)
charm-helpers-hooks.yaml (+13/-0)
charm-helpers-tests.yaml (+5/-0)
config.yaml (+53/-9)
hooks/charmhelpers/contrib/hahelpers/apache.py (+10/-3)
hooks/charmhelpers/contrib/hahelpers/cluster.py (+59/-17)
hooks/charmhelpers/contrib/network/ip.py (+343/-0)
hooks/charmhelpers/contrib/openstack/amulet/deployment.py (+94/-0)
hooks/charmhelpers/contrib/openstack/amulet/utils.py (+276/-0)
hooks/charmhelpers/contrib/openstack/context.py (+237/-59)
hooks/charmhelpers/contrib/openstack/ip.py (+79/-0)
hooks/charmhelpers/contrib/openstack/neutron.py (+14/-0)
hooks/charmhelpers/contrib/openstack/templates/haproxy.cfg (+18/-4)
hooks/charmhelpers/contrib/openstack/templates/openstack_https_frontend (+9/-8)
hooks/charmhelpers/contrib/openstack/templates/openstack_https_frontend.conf (+9/-8)
hooks/charmhelpers/contrib/openstack/templating.py (+22/-23)
hooks/charmhelpers/contrib/openstack/utils.py (+45/-6)
hooks/charmhelpers/contrib/peerstorage/__init__.py (+131/-0)
hooks/charmhelpers/contrib/storage/linux/ceph.py (+1/-1)
hooks/charmhelpers/core/fstab.py (+116/-0)
hooks/charmhelpers/core/hookenv.py (+49/-21)
hooks/charmhelpers/core/host.py (+75/-11)
hooks/charmhelpers/core/services/__init__.py (+2/-0)
hooks/charmhelpers/core/services/base.py (+313/-0)
hooks/charmhelpers/core/services/helpers.py (+239/-0)
hooks/charmhelpers/core/templating.py (+51/-0)
hooks/charmhelpers/fetch/__init__.py (+115/-32)
hooks/charmhelpers/fetch/archiveurl.py (+49/-4)
hooks/swift_context.py (+30/-11)
hooks/swift_hooks.py (+78/-23)
hooks/swift_utils.py (+26/-3)
revision (+1/-1)
templates/essex/proxy-server.conf (+2/-0)
templates/grizzly/proxy-server.conf (+2/-0)
templates/havana/proxy-server.conf (+2/-0)
templates/icehouse/proxy-server.conf (+4/-1)
templates/memcached.conf (+1/-1)
tests/00-setup (+11/-0)
tests/10-basic-precise-essex (+9/-0)
tests/11-basic-precise-folsom (+11/-0)
tests/12-basic-precise-grizzly (+11/-0)
tests/13-basic-precise-havana (+11/-0)
tests/14-basic-precise-icehouse (+11/-0)
tests/15-basic-trusty-icehouse (+9/-0)
tests/README (+52/-0)
tests/basic_deployment.py (+827/-0)
tests/charmhelpers/contrib/amulet/deployment.py (+72/-0)
tests/charmhelpers/contrib/amulet/utils.py (+176/-0)
tests/charmhelpers/contrib/openstack/amulet/deployment.py (+94/-0)
tests/charmhelpers/contrib/openstack/amulet/utils.py (+276/-0)
unit_tests/test_templates.py (+2/-2)
Revision history for this message
| Alexander List (alexlist) wrote | #1 |
| description: | updated |
| tags: | added: canonical-bootstack |
| tags: | added: openstack |
Revision history for this message
| JuanJo Ciarlante (jjo) wrote | #2 |
Revision history for this message
| JuanJo Ciarlante (jjo) wrote | #3 |
| Changed in swift-proxy (Juju Charms Collection): | |
| status: | New → In Progress |
| assignee: | nobody → JuanJo Ciarlante (jjo) |
| Changed in swift-proxy (Juju Charms Collection): | |
| importance: | Undecided → High |
| Changed in swift-proxy (Juju Charms Collection): | |
| status: | In Progress → Fix Committed |
| Changed in swift-proxy (Juju Charms Collection): | |
| status: | Fix Committed → Fix Released |
To post a comment you must log in.
I was facing the same issue because I was missing 'swift-hash' to
swift-proxy juju config, adding it fixed the issue for me.
Nevertheless IMO randomly initiaziling each unit separately is an arguable
feature, specially since we can derive a deterministic per juju env UUID
from it + swift-proxy service name: please see this MP[0],
which in short does:
swift_hash = UUID($JUJU_
[0] https:/