Keystone DB gets all non vip endpoints + openstack service conf files get keystone non vip

Bug #1508575 reported by james beedy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Keystone Charm
Fix Released
High
Edward Hope-Morley

Bug Description

After battling with HA deploys for a while now, I have found the root cause(s) of my problems.... they are such that keystone creates non vip endpoints in the keystone.endpoints table for each openstack service to which it is related, AND all openstack service charms get non vip keystone endpoints transcluded into their .conf files.

I'm deploying all next branch charms minus glance-simplestreams-sync.

juju --version <- 1.24.6-trusty-amd64

juju status --format tabular <- http://paste.ubuntu.com/12887639/

deployer.yaml <- http://paste.ubuntu.com/12887642/

It would be nice to see this fixed before the 15.10 release so as we can move forward into the next cycle with more stable HA deployment capability!

Thanks!

Revision history for this message
James Page (james-page) wrote :

"they are such that keystone creates non vip endpoints in the keystone.endpoints table for each openstack service to which it is related, AND all openstack service charms get non vip keystone endpoints transcluded into their .conf files."

This would indicate that clustered never successfully completed; the principle charms won't re-registered VIP's until the hacluster subordinates have indicated that clustering has been completed.

A 'sudo crm status' on any of the haclustered' services would be useful at this point as well.

Revision history for this message
james beedy (jamesbeedy) wrote :

To note: Initially 1 of the keystone endpoints (the admin endpoint of http://10.16.100.34:35357/v2.0 ) was set correctly to the vip in the database, every other endpoint was not set to the vip including the other keystone endpoints.

Revision history for this message
james beedy (jamesbeedy) wrote :

Here is the keystone.endpoint db table from a recent deploy: http://paste.ubuntu.com/12890703/

trusty-kilo-ha-conf.yaml: http://paste.ubuntu.com/12891151/

One identification I can make here, is that ceph-radosgw endpoint is the only endpoint set to a vip. Interestingly enough, radosgw is the only stateless service charm that still uses the old-school way of setting its endpoint ...in the sense that it does not have the same params checking and creation methods that the rest of the charms do within the ha-relation-{joined|changed} hooks.

One workaround for this issue could be to set os-public-endpoint params for each charm....unfortunately it does not solve the issue.

Revision history for this message
james beedy (jamesbeedy) wrote :

^^ trusty-kilo-conf.yaml should be: http://paste.ubuntu.com/12891514/

Revision history for this message
james beedy (jamesbeedy) wrote :

I feel this is actually more of a conceptual misunderstanding. After pondering this issue for a while now, it dawned on me.....

You cannot create a VIP on a bridge interface.
You cannot create a VIP on an interface to which a bridge is bound.

The former, is due to the fact that the bridge owns the interface to which it is bound, the later is because the bridge is already a virtual device.

1. Using MAAS as a provider, 'juju-br0' is created on the primary interface ('eth0' or last interface provisioned with gateway route) for physical/virtual nodes.
2. Juju + MAAS provisioned containers only have 1 network interface and it gets owned by 'juju-br0'.

  2.a. 'juju-br0' owns the interface it is bridged on (no vips can be created on 'eth0' when 'juju-br0' is bound to 'eth0').
  2.b. You can't create a VIP on a virtual bridge interface (i.e. 'juju-br0')!
  2.c. Containers provisioned by juju only have one network interface (hence no option for VIP).

I believe this proves that currently:
    - HA deployed service units must configure VIP on interface other than interface which 'juju-br0' binds to (must set 'vip_iface' and 'ha-bindiface' to interface other than the primary).
    - Services deployed to containers not eligible for HA.

Let me know your thoughts.

Thanks!

Revision history for this message
james beedy (jamesbeedy) wrote :

Extra notes:
2.c is redundant of 2.
I take back the work-around from comment #3.

Revision history for this message
Ryan Beisner (1chb1n) wrote :

Thank you for the bug report. We're working to reproduce in our bare metal lab. Once triaged, the bug will get status updates.

David Ames (thedac)
Changed in keystone (Juju Charms Collection):
assignee: nobody → David Ames (thedac)
status: New → In Progress
Revision history for this message
David Ames (thedac) wrote :

James, in your comments above, it seems we are mixing physical host network setup (juju-br0) with lxc network setup. The lxcs should not have juju-br0 in them.

juju-br0 is used on the physical host to bridge traffic for itself and all of its lxcs.

We can apply a vip (an alias) to eth0 *inside* the lxc. We are doing that for ServerStack and I just stood up an independent test in another MAAS environment.

On the physcial host there are several bridges that allow the virtual networking. But notice that eth0 is a member of juju-br0 and that the IP for the physical host is on juju-br0. This is by design and works.

Example lxc eth0
sudo ip addr

43330: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:16:3e:b0:31:b1 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.50/24 brd 10.245.167.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.0.0.200/24 brd 10.245.167.255 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:feb0:31b1/64 scope link
       valid_lft forever preferred_lft forever

In this example 10.0.0.200 is the vip and .50 is the main ip for the lxc. Corosync decides which node has the alias for the vip.

So, we have not yet been able to recreate failure to distribute keystones vip. We still need to figure out what is different in your environment.

Revision history for this message
james beedy (jamesbeedy) wrote :

thedac, this makes total sense....I'll be looking further into reproducing this and getting more info on what is really going on. I appreciate your looking into this. I'll keep you posted. Thanks

Revision history for this message
David Ames (thedac) wrote :

James,

Just an update. I spent the week trying to reproduce by deploying via MAAS using most of [1]. Unfortunately I was not able to reproduce this.

If you see this again please provide the following from the nodes that have presented non-VIP addresses to keystone
As James Page requested we would need: sudo crm status
The /var/log/juju/* logs

[1] http://paste.ubuntu.com/12891151/

Changed in keystone (Juju Charms Collection):
status: In Progress → Incomplete
Liam Young (gnuoy)
Changed in keystone (Juju Charms Collection):
importance: Undecided → Medium
James Page (james-page)
Changed in charm-keystone:
assignee: nobody → David Ames (thedac)
importance: Undecided → Medium
status: New → Incomplete
Changed in keystone (Juju Charms Collection):
status: Incomplete → Invalid
Revision history for this message
Edward Hope-Morley (hopem) wrote :

I've hit a very similar issue with the next charms whereby:

  * keystone has 3 units
  * keystone has a vip configured
  * keystone relations all receive service_host set to the vip
  * keystone db (i.e. endpoint) is configured to use non-vip

Looking at the code I think I can see why this is happening. The endpoint gets created by [1] which can be called from shared-db db_changed [2] hook or config_changed hook [3].

[1] https://github.com/openstack/charm-keystone/blob/stable/18.08/hooks/keystone_utils.py#L1105
[2] https://github.com/openstack/charm-keystone/blob/stable/18.08/hooks/keystone_hooks.py#L346
[3] https://github.com/openstack/charm-keystone/blob/stable/18.08/hooks/keystone_hooks.py#L246
[4] https://github.com/openstack/charm-keystone/blob/stable/18.08/charmhelpers/contrib/openstack/ip.py#L116

The problem is that if [2] executes prior to keystone being clustered then [4] never returns the vip hence the endpoint is set to be the local address of the unit. If I trigger a config change this does get fixed by [3] BUT if i don't it never gets resolved because [2] will never call [1] again once the db is initialised. I think we can easily fix this by adding a gratuitous call to [1] once the hacluster is complete.

Changed in charm-keystone:
milestone: none → 18.11
importance: Medium → High
assignee: David Ames (thedac) → Edward Hope-Morley (hopem)
status: Incomplete → New
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/616893

Changed in charm-keystone:
status: New → In Progress
no longer affects: keystone (Juju Charms Collection)
Revision history for this message
james beedy (jamesbeedy) wrote :

@hopem this is most excellent! thank you!

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-keystone (master)

Reviewed: https://review.openstack.org/616893
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=614db19f8c5415ba5f9fe88175d21756dc2220c7
Submitter: Zuul
Branch: master

commit 614db19f8c5415ba5f9fe88175d21756dc2220c7
Author: Edward Hope-Morley <email address hidden>
Date: Fri Nov 9 12:15:22 2018 +0000

    Ensure endpoint up-to-date once ha completes

    Change-Id: I5f6f7c7f0acb1b7951730c02a55e4971f00d5c9d
    Closes-Bug: #1508575

Changed in charm-keystone:
status: In Progress → Fix Committed
David Ames (thedac)
Changed in charm-keystone:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.