[SRU] crm_simulate segfaults

[nunogt]

[Impact]

Low to Medium impact. crm_simulate is used to simulate changes that will be introduced to the cluster. Not being able to use means one can't check if upcoming changes will break anything.

[Test Case]

$ sudo apt-get install pacemaker
$ sudo crm_simulate -L

With current precise packages, this will cause segfault.

[Regression potential]

This is an upstream commit that's part of the package in Raring and Saucy. By looking at the code, this shouldn't introduce any regression at all.

Original report:

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04.1 LTS
Release: 12.04
Codename: precise

pacemaker:
  Installed: 1.1.6-2ubuntu3
  Candidate: 1.1.6-2ubuntu3
  Version table:
 *** 1.1.6-2ubuntu3 0
        500 http://pt.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
        100 /var/lib/dpkg/status

Using crm_simulate -SL would, according to the manpage, simulate the transition's execution and display the resulting cluster status.

Instead, I get the following:
# crm_simulate -SL
Segmentation fault (core dumped)

[Jacob Smith (jsmith-argotecinc)]

This is a known bug in Pacemaker version 1.1.6

I understand that with the LTS release there may be no hope of getting the package upgraded to 1.1.7 but we should at least be able to get this bug fixed I would hope.

Patch to fix this is here:
https://github.com/beekhof/pacemaker/commit/2bdb230

Output of $crm_simulate -L:

*** glibc detected *** crm_simulate: double free or corruption (out): 0x0000000001585e00 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7e626)[0x7fef66e82626]
/usr/lib/libcib.so.1(+0xc259)[0x7fef6775f259]
/lib/x86_64-linux-gnu/libglib-2.0.so.0(+0x373d3)[0x7fef66b463d3]
/lib/x86_64-linux-gnu/libglib-2.0.so.0(g_hash_table_remove_all+0x1d)[0x7fef66b4724d]
/lib/x86_64-linux-gnu/libglib-2.0.so.0(g_hash_table_destroy+0xe)[0x7fef66b472de]
/usr/lib/libcib.so.1(cib_new_variant+0x155)[0x7fef6775f9fb]
/usr/lib/libcib.so.1(cib_file_new+0x28)[0x7fef67765aad]
/usr/lib/libcib.so.1(cib_new+0x62)[0x7fef6775f70c]
crm_simulate[0x40614f]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7fef66e2576d]
crm_simulate[0x402279]
======= Memory map: ========
00400000-00409000 r-xp 00000000 fb:05 541771 /usr/sbin/crm_simulate
00608000-00609000 r--p 00008000 fb:05 541771 /usr/sbin/crm_simulate
00609000-0060a000 rw-p 00009000 fb:05 541771 /usr/sbin/crm_simulate
0157e000-0181c000 rw-p 00000000 00:00 0 [heap]
7fef6399a000-7fef639af000 r-xp 00000000 fb:05 393311 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fef639af000-7fef63bae000 ---p 00015000 fb:05 393311 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fef63bae000-7fef63baf000 r--p 00014000 fb:05 393311 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fef63baf000-7fef63bb0000 rw-p 00015000 fb:05 393311 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fef63bb0000-7fef63bb3000 r-xp 00000000 fb:05 393398 /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
7fef63bb3000-7fef63db2000 ---p 00003000 fb:05 393398 /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
7fef63db2000-7fef63db3000 r--p 00002000 fb:05 393398 /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
7fef63db3000-7fef63db4000 rw-p 00003000 fb:05 393398 /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
7fef63db4000-7fef63dbc000 r-xp 00000000 fb:05 525256 /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
7fef63dbc000-7fef63fbc000 ---p 00008000 fb:05 525256 /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
7fef63fbc000-7fef63fbd000 r--p 00008000 fb:05 525256 /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
7fef63fbd000-7fef63fbe000 rw-p 00009000 fb:05 525256 /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
7fef63fbe000-7fef63fcf000 r-xp 00000000 fb:05 525504 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
7fef63fcf000-7fef641ce000 ---p 00011000 fb:05 525504 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
7fef641ce000-7fef641cf000 r--p 00010000 fb:05 525504 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
7fef641cf0...

[Luca Lorenzetto (lorenzetto-luca)]

I have the same bug. On syslog i find:

Jan 8 09:00:00 posta1204-1 kernel: [77036.289124] crm_simulate[23625]: segfault at 20 ip 00007fd2c8b41207 sp 00007fffe7aeb820 error 4 in libcib.so.1.0.1[7fd2c8b35000+1b000]

I think fixing this bug will be a very good idea.

[Ante Karamatić (ivoks)]

This patch solves the problem.

[Colin Watson (cjwatson)]

Hello nunogt, or anyone else affected,

Accepted pacemaker into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/pacemaker/1.1.6-2ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[darren (darrenm)]

This didn't fix it for me:

 crm_simulate -L -VVVVV
crm_simulate[2873]: 2013/07/24_10:41:40 debug: init_client_ipc_comms_nodispatch: Attempting to talk on: /var/run/crm/cib_rw
crm_simulate[2873]: 2013/07/24_10:41:41 debug: init_client_ipc_comms_nodispatch: Attempting to talk on: /var/run/crm/cib_callback
crm_simulate[2873]: 2013/07/24_10:41:41 debug: cib_native_signon_raw: Connection to CIB successful
crm_simulate[2873]: 2013/07/24_10:41:41 debug: cib_native_signoff: Signing out of the CIB Service
crm_simulate[2873]: 2013/07/24_10:41:41 info: validate_with_relaxng: Creating RNG parser context
Segmentation fault (core dumped)

dpkg-query -l pacemaker
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-=============================================================-=============================================================-==========================================================================================================================================
ii pacemaker 1.1.6-2ubuntu3.1 HA cluster resource manager

[Ante Karamatić (ivoks)]

@darren make sure you also update 'libcib1' package. That's the binary where the fix is contained.

[Ante Karamatić (ivoks)]

# crm_simulate -SL -VVVV
crm_simulate[9934]: 2013/07/29_10:24:54 debug: init_client_ipc_comms_nodispatch: Attempting to talk on: /var/run/crm/cib_rw
crm_simulate[9934]: 2013/07/29_10:24:54 debug: init_client_ipc_comms_nodispatch: Attempting to talk on: /var/run/crm/cib_callback
crm_simulate[9934]: 2013/07/29_10:24:54 debug: cib_native_signon_raw: Connection to CIB successful
crm_simulate[9934]: 2013/07/29_10:24:54 debug: cib_native_signoff: Signing out of the CIB Service
crm_simulate[9934]: 2013/07/29_10:24:54 info: validate_with_relaxng: Creating RNG parser context
crm_simulate[9934]: 2013/07/29_10:24:54 debug: cib_file_signon: crm_simulate: Opened connection to local file '/var/lib/heartbeat/crm/shadow.9934'
crm_simulate[9934]: 2013/07/29_10:24:54 info: cib_file_perform_op_delegate: cib_query on (null)
crm_simulate[9934]: 2013/07/29_10:24:54 info: unpack_config: Startup probes: enabled
crm_simulate[9934]: 2013/07/29_10:24:54 debug: unpack_config: STONITH timeout: 60000
crm_simulate[9934]: 2013/07/29_10:24:54 debug: unpack_config: STONITH of failed nodes is enabled
crm_simulate[9934]: 2013/07/29_10:24:54 debug: unpack_config: Stop all active resources: false
crm_simulate[9934]: 2013/07/29_10:24:54 debug: unpack_config: Cluster is symmetric - resources can run anywhere by default
crm_simulate[9934]: 2013/07/29_10:24:54 debug: unpack_config: Default stickiness: 0
crm_simulate[9934]: 2013/07/29_10:24:54 debug: unpack_config: On loss of CCM Quorum: Stop ALL resources
crm_simulate[9934]: 2013/07/29_10:24:54 info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
crm_simulate[9934]: 2013/07/29_10:24:54 WARN: cluster_status: We do not have quorum - fencing and resource management disabled
crm_simulate[9934]: 2013/07/29_10:24:54 info: unpack_domains: Unpacking domains
crm_simulate[9934]: 2013/07/29_10:24:54 ERROR: unpack_resources: Resource start-up disabled since no STONITH resources have been defined
crm_simulate[9934]: 2013/07/29_10:24:54 ERROR: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option
crm_simulate[9934]: 2013/07/29_10:24:54 ERROR: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity
crm_simulate[9934]: 2013/07/29_10:24:54 info: determine_online_status: Node jupiter is online

Current cluster status:
Online: [ jupiter ]

crm_simulate[9934]: 2013/07/29_10:24:54 info: set_graph_functions: Setting custom graph functions

Executing cluster transition:
crm_simulate[9934]: 2013/07/29_10:24:54 info: unpack_graph: Unpacked transition 0: 0 actions in 0 synapses
crm_simulate[9934]: 2013/07/29_10:24:54 info: run_graph: ====================================================
crm_simulate[9934]: 2013/07/29_10:24:54 notice: run_graph: Transition 0 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=crm_simulate): Complete
crm_simulate[9934]: 2013/07/29_10:24:54 info: cib_file_perform_op_delegate: cib_query on (null)

Revised cluster status:
crm_simulate[9934]: 2013/07/29_10:24:54 info: unpack_config: Startup probes: enabled
crm_simulate[9934]: 2013/07/29_10:24:54 debug: unpack_config: STONITH timeout: 60000
crm_simulate[9934]: 2013/07/29_10:24...

[Launchpad Janitor (janitor)]

This bug was fixed in the package pacemaker - 1.1.6-2ubuntu3.1

---------------
pacemaker (1.1.6-2ubuntu3.1) precise-proposed; urgency=low

  * debian/patches/cib-segfault.patch: Fix segfault caused by
    destroying an already free hash table. (LP: #1057651)
 -- Ante Karamatic <email address hidden> Thu, 27 Jun 2013 15:55:34 +0200

[Colin Watson (cjwatson)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.