HA failure when no IP address is bound to the VIP interface
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Swift Proxy Charm |
Fix Released
|
High
|
James Page | ||
cinder (Juju Charms Collection) |
Fix Released
|
High
|
James Page | ||
glance (Juju Charms Collection) |
Fix Released
|
High
|
James Page | ||
keystone (Juju Charms Collection) |
Fix Released
|
High
|
James Page | ||
neutron-api (Juju Charms Collection) |
Fix Released
|
High
|
James Page | ||
nova-cloud-controller (Juju Charms Collection) |
Fix Released
|
High
|
James Page | ||
openstack-dashboard (Juju Charms Collection) |
Fix Released
|
High
|
James Page | ||
percona-cluster (Juju Charms Collection) |
Invalid
|
High
|
Unassigned | ||
swift-proxy (Juju Charms Collection) |
Invalid
|
High
|
James Page |
Bug Description
Proxying from juju ML:
We've been working on setting up an Openstack cluster on Trusty for a few months now using Juju and MAAS, although we've yet to go into production. I had everything working fine, including HA deployments of Keystone, Glance, Percona etc.
The older versions of the charms supported HA using the config settings vip, vip_cidr and vip_iface. Without me making any modifications to these charms, I successfully deployed all of the above charms with the bog-standard hacluster charm.
Over the weekend I've been updating to Juno, and I naturally updated to the latest stable charms from the Charm store. Breaking changes have been introduced to these charms such that they no longer support my deployment. My Openstack cluster promptly broke in a nasty way. I'm *really* glad this isn't a production environment, but these kinds of non-backward compatible breakages do give me cause for concern going forward.
To explain how this broke, I'll first need to explain how our network was deployed:
In order to not burn through many public IPs, we assign RFC1918 IPs to *every server* by DHCP.
We run at least two instances of critical services
Public IPs are assigned primarily by Pacemaker
Public and Private subnets coexist on a single Layer-2 network.
Nodes that do not directly participate in the Public subnet still have direct access (not via a router) to the Public IPs courtesy of the DHCP option (rfc3442-
This set-up was highly efficient in terms of consumption of valuable public IP addresses, without forcing inter-subnet communications via an unnecessary hop. The only trick that we had to pull-off was getting the DHCP server to give out the rfc3442-
The old OpenStack charms with their simple vip, vip_cidr and vip_iface options worked perfectly with this set-up. The new charms cannot support this at all, as they have become, in my view, "too clever". They now insist that the vip can only be bound to an interface that already has an IP in the same subnet.
If I have to bind public IPs to every server (IPs that they will never use) just in order to have Pacemaker assign the vip, I'll burn through a lot of IPs in the most pointless way imaginable.
I've modified the keystone and openstack-dashboard charms to re-introduce the old functionality in a way that doesn't break the new multiple-IP functionality. I'll paste my keystone patch below to give you an idea what I think is needed. This hasn't been thoroughly tested, but it seems to work. Pacemaker can at least set the public IP address again.
If there is some other (better) way to achieve the same level of IP address allocation efficiency and performance without patching the Openstack charms, please point me in the right direction.
Thanks,
John
Related branches
- Ryan Beisner (community): Approve
- Liam Young (community): Approve
-
Diff: 98 lines (+57/-2)3 files modifiedconfig.yaml (+12/-0)
hooks/cinder_hooks.py (+6/-2)
unit_tests/test_cluster_hooks.py (+39/-0)
- Ryan Beisner (community): Approve
- Liam Young (community): Approve
-
Diff: 87 lines (+46/-2)3 files modifiedconfig.yaml (+12/-0)
hooks/glance_relations.py (+6/-2)
unit_tests/test_glance_relations.py (+28/-0)
- Ryan Beisner (community): Approve
- Liam Young (community): Approve
-
Diff: 84 lines (+43/-2)3 files modifiedconfig.yaml (+12/-0)
hooks/keystone_hooks.py (+6/-2)
unit_tests/test_keystone_hooks.py (+25/-0)
- Liam Young (community): Approve
-
Diff: 91 lines (+50/-2)3 files modifiedconfig.yaml (+12/-0)
hooks/neutron_api_hooks.py (+6/-2)
unit_tests/test_neutron_api_hooks.py (+32/-0)
- Ryan Beisner (community): Approve
- Liam Young (community): Approve
-
Diff: 94 lines (+49/-2)3 files modifiedconfig.yaml (+12/-0)
hooks/nova_cc_hooks.py (+6/-2)
unit_tests/test_nova_cc_hooks.py (+31/-0)
- Ryan Beisner (community): Approve
- Liam Young (community): Approve
-
Diff: 89 lines (+48/-2)3 files modifiedconfig.yaml (+12/-0)
hooks/horizon_hooks.py (+6/-2)
unit_tests/test_horizon_hooks.py (+30/-0)
- Liam Young (community): Approve
-
Diff: 112 lines (+34/-15)3 files modifiedcharmhelpers/contrib/openstack/context.py (+18/-14)
charmhelpers/contrib/openstack/templates/haproxy.cfg (+3/-1)
tests/contrib/openstack/test_os_contexts.py (+13/-0)
tags: | added: openstack |
Changed in nova-cloud-controller (Juju Charms Collection): | |
milestone: | none → 15.01 |
Changed in cinder (Juju Charms Collection): | |
milestone: | none → 15.01 |
Changed in glance (Juju Charms Collection): | |
milestone: | none → 15.01 |
Changed in keystone (Juju Charms Collection): | |
milestone: | none → 15.01 |
Changed in openstack-dashboard (Juju Charms Collection): | |
milestone: | none → 15.01 |
Changed in neutron-api (Juju Charms Collection): | |
milestone: | none → 15.01 |
Changed in cinder (Juju Charms Collection): | |
assignee: | nobody → James Page (james-page) |
Changed in swift (Ubuntu): | |
assignee: | nobody → James Page (james-page) |
Changed in glance (Juju Charms Collection): | |
assignee: | nobody → James Page (james-page) |
Changed in keystone (Juju Charms Collection): | |
assignee: | nobody → James Page (james-page) |
Changed in neutron-api (Juju Charms Collection): | |
assignee: | nobody → James Page (james-page) |
Changed in nova-cloud-controller (Juju Charms Collection): | |
assignee: | nobody → James Page (james-page) |
Changed in openstack-dashboard (Juju Charms Collection): | |
assignee: | nobody → James Page (james-page) |
affects: | swift (Ubuntu) → swift-proxy (Juju Charms Collection) |
Changed in swift-proxy (Juju Charms Collection): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in nova-cloud-controller (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in cinder (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in glance (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in keystone (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in openstack-dashboard (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in neutron-api (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in charm-swift-proxy: | |
assignee: | nobody → James Page (james-page) |
importance: | Undecided → High |
status: | New → Triaged |
Changed in swift-proxy (Juju Charms Collection): | |
status: | Triaged → Invalid |
After some discussion on IRC, we're going to re-use the previous configuration options vip_iface and vip_cidr so that any upgraders also using John's configuration should just DTRT.