nova-cloud-controller hacluster VIP issues (Updated CIB does not validate against pacemaker-2.4 schema/dtd)

Bug #1732785 reported by Mathieu Aubert
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Nova Cloud Controller Charm
Confirmed
Undecided
Unassigned

Bug Description

On some occasion (10% of our deployment) the nova-cloud-controller not being able to setup its hacluster VIP.

Running the command: crm configure show

Shows that the primitive for res_nova_eth0_vip, res_nova_eth1_vip and res_nova_haproxy are not configured.

If we remove the nova-cloud-controller units from the machines and add them again to VIP will be configure this time.

Is there a reason for this behavior?

We included logs and commands outputs from a bad and a good system for comparison.

Revision history for this message
Mathieu Aubert (maubert43) wrote :
Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :
Download full text (5.0 KiB)

This is newton openstack AFAIKs.

var/log/juju/unit-hacluster-nova-cloud-controller-0.log:2017-11-16 12:18:10 DEBUG install Get:49 http://archive.ubuntu.com/ubuntu xenial/universe ...

Which charm release though: 17.02, 17.08 or something else? Could you please paste the relevant "juju status" with charm revision numbers? Otherwise it's hard to say where this may be coming from.

On a "bad" hacluster there's a CIB rendered that doesn't pass DTD validation.

Updated CIB does not validate against pacemaker-2.4 schema/dtd

See below:

hacluster unit log (bad):
Nov 16 12:22:53 [169497] juju-37c0a4-1-lxd-4 cib: warning: cib_process_request: Completed cib_apply_diff operation for section 'all': Update does not conform to the configured schema (rc=-203, origin=juju-37c0a4-2-lxd-4/cibadmin/2, version=0.8.16)
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: Diff: --- 0.8.16 2
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: Diff: +++ 0.9.0 (null)
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: + /cib: @epoch=9, @num_updates=0
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: ++ /cib/configuration/resources: <group id="grp_nova_vips"/>
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Expecting an element meta_attributes, got nothing
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Invalid sequence in interleave
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Element group failed to validate content
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Element resources has extra content: group
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: warning: cib_perform_op: Updated CIB does not validate against pacemaker-2.4 schema/dtd
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: warning: cib_diff_notify: Update (client: cibadmin, call:2): 0.8.16 -> 0.9.0 (Update does not conform to the configured schema)
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: warning: cib_process_request: Completed cib_apply_diff operation for section 'all': Update does not conform to the configured schema (rc=-203, origin=juju-37c0a4-0-lxd-4/cibadmin/2, version=0.8.16)
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: Diff: --- 0.8.16 2
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: Diff: +++ 0.9.0 (null)
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: + /cib: @epoch=9, @num_updates=0
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: ++ /cib/configuration/resources: <clone id="cl_nova_haproxy"/>
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Expecting an element meta_attributes, got nothing
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Invalid sequence in interleave
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_...

Read more...

Changed in charm-nova-cloud-controller:
status: New → Incomplete
summary: - nova-cloud-controller hacluster VIP issues
+ nova-cloud-controller hacluster VIP issues (Updated CIB does not
+ validate against pacemaker-2.4 schema/dtd)
Revision history for this message
Bertin Colpron (bcolpron) wrote :
Download full text (3.8 KiB)

Let me answer that one on behalf of Mathieu:

Yes, this is OpenStack Newton using charms 17.08 charms.

Here is the Juju status applications section:

App Version Status Scale Charm Store Rev OS Notes
ceilometer 7.1.1 waiting 3 ceilometer jujucharms 248 ubuntu
ceilometer-agent 7.1.1 waiting 3 ceilometer-agent jujucharms 239 ubuntu
ceph-mon 10.2.9 active 3 ceph-mon jujucharms 13 ubuntu
ceph-osd 10.2.9 active 3 ceph-osd jujucharms 248 ubuntu
ceph-radosgw 10.2.9 waiting 3 ceph-radosgw jujucharms 252 ubuntu
cinder 9.1.4 waiting 3 cinder jujucharms 264 ubuntu
cinder-backup 9.1.4 blocked 2/3 cinder-backup jujucharms 8 ubuntu
cinder-ceph 9.1.4 active 3 cinder-ceph jujucharms 227 ubuntu
glance 13.0.0 waiting 3 glance jujucharms 259 ubuntu
hacluster-ceilometer active 3 hacluster jujucharms 37 ubuntu
hacluster-ceph-radosgw active 3 hacluster jujucharms 37 ubuntu
hacluster-cinder active 3 hacluster jujucharms 37 ubuntu
hacluster-glance active 3 hacluster jujucharms 37 ubuntu
hacluster-heat active 3 hacluster jujucharms 37 ubuntu
hacluster-horizon active 3 hacluster jujucharms 37 ubuntu
hacluster-keystone waiting 0/3 hacluster jujucharms 37 ubuntu
hacluster-mysql active 3 hacluster jujucharms 37 ubuntu
hacluster-neutron-api active 3 hacluster jujucharms 37 ubuntu
hacluster-nova-cloud-controller waiting 0/3 hacluster jujucharms 37 ubuntu
heat 7.0.6 waiting 3 heat jujucharms 244 ubuntu
keystone 10.0.3 active 3 keystone jujucharms 271 ubuntu
memcached unknown 3 memcached jujucharms 18 ubuntu
mongodb unknown 3 mongodb jujucharms 46 ubuntu
mysql 5.6.34-26.19 active 3 percona-cluster jujucharms 255 ubuntu
neutron-api 9.4.0 waiting 3 neutron-api jujucharms 253 ubuntu
neutron-gateway 9.4.0 waiting 2 neutron-gateway jujucharms 241 ubuntu
neutron-openvswitch 9.4.0 active 3 neutron-...

Read more...

Changed in charm-nova-cloud-controller:
status: Incomplete → Confirmed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.