Which charm release though: 17.02, 17.08 or something else? Could you please paste the relevant "juju status" with charm revision numbers? Otherwise it's hard to say where this may be coming from.
On a "bad" hacluster there's a CIB rendered that doesn't pass DTD validation.
Updated CIB does not validate against pacemaker-2.4 schema/dtd
See below:
hacluster unit log (bad):
Nov 16 12:22:53 [169497] juju-37c0a4-1-lxd-4 cib: warning: cib_process_request: Completed cib_apply_diff operation for section 'all': Update does not conform to the configured schema (rc=-203, origin=juju-37c0a4-2-lxd-4/cibadmin/2, version=0.8.16)
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: Diff: --- 0.8.16 2
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: Diff: +++ 0.9.0 (null)
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: + /cib: @epoch=9, @num_updates=0
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: ++ /cib/configuration/resources: <group id="grp_nova_vips"/>
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Expecting an element meta_attributes, got nothing
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Invalid sequence in interleave
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Element group failed to validate content
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Element resources has extra content: group
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: warning: cib_perform_op: Updated CIB does not validate against pacemaker-2.4 schema/dtd
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: warning: cib_diff_notify: Update (client: cibadmin, call:2): 0.8.16 -> 0.9.0 (Update does not conform to the configured schema)
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: warning: cib_process_request: Completed cib_apply_diff operation for section 'all': Update does not conform to the configured schema (rc=-203, origin=juju-37c0a4-0-lxd-4/cibadmin/2, version=0.8.16)
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: Diff: --- 0.8.16 2
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: Diff: +++ 0.9.0 (null)
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: + /cib: @epoch=9, @num_updates=0
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: ++ /cib/configuration/resources: <clone id="cl_nova_haproxy"/>
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Expecting an element meta_attributes, got nothing
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Invalid sequence in interleave
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Element clone failed to validate content
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Element resources has extra content: clone
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: warning: cib_perform_op: Updated CIB does not validate against pacemaker-2.4 schema/dtd
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: warning: cib_diff_notify: Update (client: cibadmin, call:2): 0.8.16 -> 0.9.0 (Update does not conform to the configured schema)
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: warning: cib_process_request: Completed cib_apply_diff operation for section 'all': Update does not conform to the configured schema (rc=-203, origin=juju-37c0a4-0-lxd-4/cibadmin/2, version=0.8.16)
Nov 16 12:36:09 [169502] juju-37c0a4-1-lxd-4 crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
nova-cloud-controller unit (bad):
017-11-16 12:22:38 DEBUG juju-log ha:86: Configuring Groups: {'grp_nova_vips': 'res_nova_eth0_vip res_nova_eth1_vip'}
2017-11-16 12:22:38 DEBUG ha-relation-changed ERROR: grp_nova_vips refers to missing object res_nova_eth0_vip
2017-11-16 12:22:38 DEBUG ha-relation-changed ERROR: grp_nova_vips refers to missing object res_nova_eth1_vip
2017-11-16 12:22:38 DEBUG ha-relation-changed ERROR: syntax in group: Expected at least 2 arguments near <group> parsing 'group grp_nova_vips'
2017-11-16 12:22:38 DEBUG ha-relation-changed Call cib_apply_diff failed (-203): Update does not conform to the configured schema
2017-11-16 12:22:38 DEBUG ha-relation-changed <cib crm_feature_set="3.0.10" validate-with="pacemaker-2.4" epoch="9" num_updates="0" admin_epoch="0" cib-last-written="Thu Nov 16 12:22:38 2017" update-origin="juju-37c0a4-1-lxd-4" update-client="cibadmin" update-user="root" have-quorum="1" dc-uuid="1000">
This is newton openstack AFAIKs.
var/log/ juju/unit- hacluster- nova-cloud- controller- 0.log:2017- 11-16 12:18:10 DEBUG install Get:49 http:// archive. ubuntu. com/ubuntu xenial/universe ...
Which charm release though: 17.02, 17.08 or something else? Could you please paste the relevant "juju status" with charm revision numbers? Otherwise it's hard to say where this may be coming from.
On a "bad" hacluster there's a CIB rendered that doesn't pass DTD validation.
Updated CIB does not validate against pacemaker-2.4 schema/dtd
See below:
hacluster unit log (bad): request: Completed cib_apply_diff operation for section 'all': Update does not conform to the configured schema (rc=-203, origin= juju-37c0a4- 2-lxd-4/ cibadmin/ 2, version=0.8.16) ion/resources: <group id="grp_ nova_vips" /> request: Completed cib_apply_diff operation for section 'all': Update does not conform to the configured schema (rc=-203, origin= juju-37c0a4- 0-lxd-4/ cibadmin/ 2, version=0.8.16) ion/resources: <clone id="cl_ nova_haproxy" /> request: Completed cib_apply_diff operation for section 'all': Update does not conform to the configured schema (rc=-203, origin= juju-37c0a4- 0-lxd-4/ cibadmin/ 2, version=0.8.16) transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_ TIMER_POPPED origin= crm_timer_ popped ]
Nov 16 12:22:53 [169497] juju-37c0a4-1-lxd-4 cib: warning: cib_process_
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: Diff: --- 0.8.16 2
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: Diff: +++ 0.9.0 (null)
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: + /cib: @epoch=9, @num_updates=0
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: ++ /cib/configurat
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Expecting an element meta_attributes, got nothing
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Invalid sequence in interleave
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Element group failed to validate content
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Element resources has extra content: group
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: warning: cib_perform_op: Updated CIB does not validate against pacemaker-2.4 schema/dtd
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: warning: cib_diff_notify: Update (client: cibadmin, call:2): 0.8.16 -> 0.9.0 (Update does not conform to the configured schema)
Nov 16 12:23:09 [169497] juju-37c0a4-1-lxd-4 cib: warning: cib_process_
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: Diff: --- 0.8.16 2
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: Diff: +++ 0.9.0 (null)
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: + /cib: @epoch=9, @num_updates=0
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: info: cib_perform_op: ++ /cib/configurat
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Expecting an element meta_attributes, got nothing
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Invalid sequence in interleave
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Element clone failed to validate content
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: error: xml_log: Element resources has extra content: clone
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: warning: cib_perform_op: Updated CIB does not validate against pacemaker-2.4 schema/dtd
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: warning: cib_diff_notify: Update (client: cibadmin, call:2): 0.8.16 -> 0.9.0 (Update does not conform to the configured schema)
Nov 16 12:23:10 [169497] juju-37c0a4-1-lxd-4 cib: warning: cib_process_
Nov 16 12:36:09 [169502] juju-37c0a4-1-lxd-4 crmd: notice: do_state_
nova-cloud- controller unit (bad):
017-11-16 12:22:38 DEBUG juju-log ha:86: Configuring Groups: {'grp_nova_vips': 'res_nova_eth0_vip res_nova_eth1_vip'} set="3. 0.10" validate- with="pacemaker -2.4" epoch="9" num_updates="0" admin_epoch="0" cib-last- written= "Thu Nov 16 12:22:38 2017" update- origin= "juju-37c0a4- 1-lxd-4" update- client= "cibadmin" update-user="root" have-quorum="1" dc-uuid="1000">
2017-11-16 12:22:38 DEBUG ha-relation-changed ERROR: grp_nova_vips refers to missing object res_nova_eth0_vip
2017-11-16 12:22:38 DEBUG ha-relation-changed ERROR: grp_nova_vips refers to missing object res_nova_eth1_vip
2017-11-16 12:22:38 DEBUG ha-relation-changed ERROR: syntax in group: Expected at least 2 arguments near <group> parsing 'group grp_nova_vips'
2017-11-16 12:22:38 DEBUG ha-relation-changed Call cib_apply_diff failed (-203): Update does not conform to the configured schema
2017-11-16 12:22:38 DEBUG ha-relation-changed <cib crm_feature_