HAproxy goes through crm vip setup and is restarted every 5 minutes when using reactive charms

[Jason Hobbs]

grep'ing the designate hacluster unit logs for "Deleting Resources" shows the charm repeatedly going through crm vip setup:

http://paste.ubuntu.com/p/NDxDNkJwD4/

Bundle:
http://paste.ubuntu.com/p/v3YjnrYVVm/

[Tytus Kurek (tkurek)]

As discussed in bug 1737776 I do confirm I can see the same behavior in non patched environment.

This is a bug in charm-interface-hacluster. The root cause is "json_init_services" relation data being filled in with "haproxy" every time the "manage_resources" function is run.

I have a patch for it and will submit it for a review shortly.

[Tytus Kurek (tkurek)]

So it looks like the issue has been already solved by the following commit:

https://github.com/openstack/charm-interface-hacluster/commit/44c900958f9ab9d7e5c1c104b53eb651e1e444c7

[Jason Hobbs (jason-hobbs)]

We're still hitting this on -next charms as of about an hour ago.

[Tytus Kurek (tkurek)]

For me the issue is fixed when using the following charm version:

cs:~openstack-charmers-next/designate-75

which contains the patch to charm-interface-hacluster.

[Jason Hobbs (jason-hobbs)]

It may be that we are hitting separate issues, then.

On Fri, Mar 9, 2018 at 3:03 AM, Tytus Kurek <email address hidden> wrote:
> For me the issue is fixed when using the following charm version:
>
> cs:~openstack-charmers-next/designate-75
>
> which contains the patch to charm-interface-hacluster.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1754149
>
> Title:
> HAproxy goes through crm vip setup and is restarted every 5 minutes
> when using reactive charms
>
> Status in OpenStack Designate Charm:
> New
>
> Bug description:
> grep'ing the designate hacluster unit logs for "Deleting Resources"
> shows the charm repeatedly going through crm vip setup:
>
> http://paste.ubuntu.com/p/NDxDNkJwD4/
>
> Bundle:
> http://paste.ubuntu.com/p/v3YjnrYVVm/
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charm-designate/+bug/1754149/+subscriptions

[James Page (james-page)]

Marking this as Invalid - Jason if you think you have something similar please raise a new bug report with a full juju crashdump.

Thanks!

[Jason Hobbs (jason-hobbs)]

Um, I did file this as a new bug with a with a full juju crashdump. I don't get it! Why would I open another one?

[James Page (james-page)]

Charm version: cs:~openstack-charmers-next/designate-70

[James Page (james-page)]

Working to try to reproduce at tip of master.

[James Page (james-page)]

OK so this is different to the original bug in the hacluster relations for reactive charms, where services got restarted every 5 minutes (on update-status calls).

In this case the log data are someone random in timing but > 15-20 mins.

[James Page (james-page)]

Right then:

designate_2/var/log/juju/unit-hacluster-designate-0.log:2018-03-07 07:34:30 DEBUG juju-log ha:59: Configuring Groups: {u'grp_designate_vips': u'res_designate_eth0_vip res_designate_eth1_vip'}
designate_2/var/log/juju/unit-hacluster-designate-0.log:2018-03-07 08:25:58 DEBUG juju-log ha:59: Configuring Groups: {u'grp_designate_vips': u'res_designate_eth1_vip res_designate_eth0_vip'}
designate_2/var/log/juju/unit-hacluster-designate-0.log:2018-03-07 09:07:16 DEBUG juju-log ha:59: Configuring Groups: {u'grp_designate_vips': u'res_designate_eth0_vip res_designate_eth1_vip'}

The order of the vips in the group configuration for vips keeps changing - this is yet-another py3 dict iteration type problem in the hacluster interface.

[James Page (james-page)]

This will only happen in deployments with multiple VIP's

[James Page (james-page)]

https://review.openstack.org/#/c/555256/

[Corey Bryant (corey.bryant)]

I'm going to mark this as Fix Committed. The corresponding fix to charm-interface-hacluster landed on Mar 22 and charm-designate's last commit to master was Apr 2. The charm was rebuilt and I confirmed the rebuilt version is published to the charm store at https://jujucharms.com/u/openstack-charmers-next/designate/.

[Corey Bryant (corey.bryant)]

Confirmed also that James' fix is in https://jujucharms.com/u/openstack-charmers-next/designate/ at hooks/relations/hacluster/requires.py.