nova-compute process killed after libc-bin upgrade

[Brad Marshall]

nova-compute process across 2 Openstack deployments died around the same time as the recent libc-bin upgrade:

  2016-07-12 00:59:26 status half-configured libc-bin:amd64 2.19-0ubuntu6.9
  2016-07-12 00:59:26 status installed libc-bin:amd64 2.19-0ubuntu6.9

Digging into the logs showed it was possibly related to ceph:

common/ceph_crypto.cc: In function 'void ceph::crypto::init(CephContext*)' thread 7fbbdb76a740 time 2016-07-12 00:59:37.195100
common/ceph_crypto.cc: 73: FAILED assert(crypto_context != __null)
 ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)
 1: (()+0x11d2ab) [0x7fbb693b92ab]
 2: (()+0x16ad60) [0x7fbb69406d60]
 3: (()+0x1372e9) [0x7fbb693d32e9]
 4: (()+0x1347b0) [0x7fbb693d07b0]
 5: (()+0x6704f) [0x7fbb6930304f]
 6: (ffi_call_unix64()+0x4c) [0x7fbbd9447cce]
 7: (ffi_call()+0x308) [0x7fbbd9447738]
 8: (_ctypes_callproc()+0x48e) [0x7fbbd8e745fe]
 9: (()+0x15f9e) [0x7fbbd8e75f9e]
 10: (PyObject_Call()+0x36) [0x505f96]
 11: (PyEval_EvalFrameEx()+0x1dfa) [0x49b07a]
 12: (PyEval_EvalFrameEx()+0xc72) [0x499ef2]
 13: (PyEval_EvalFrameEx()+0xc72) [0x499ef2]
 14: /usr/bin/python() [0x4a1c9a]
 15: (PyObject_Call()+0x36) [0x505f96]
 16: (PyEval_EvalFrameEx()+0x1dfa) [0x49b07a]
 17: /usr/bin/python() [0x4a1c9a]
 18: (PyEval_CallObjectWithKeywords()+0xf6) [0x4dca56]
 19: (()+0x392c) [0x7fbbda44b92c]
 20: (()+0x3276) [0x7fbbda44b276]
 21: (()+0x3dc6) [0x7fbbda44bdc6]
 22: (PyEval_EvalFrameEx()+0x3259) [0x49c4d9]
 23: /usr/bin/python() [0x4a1c9a]
 24: /usr/bin/python() [0x4dfe94]
 25: (PyObject_Call()+0x36) [0x505f96]
 26: /usr/bin/python() [0x4dddad]
 27: (PyEval_EvalFrameEx()+0x965) [0x499be5]
 28: (PyEval_EvalFrameEx()+0xc72) [0x499ef2]
 29: /usr/bin/python() [0x4a1c9a]
 30: /usr/bin/python() [0x4dfe94]
 31: (PyEval_CallObjectWithKeywords()+0x6b) [0x4dc9cb]
 32: (()+0x392c) [0x7fbbda44b92c]
 33: (()+0x3276) [0x7fbbda44b276]
 34: /usr/bin/python() [0x5d9538]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
terminate called after throwing an instance of 'ceph::FailedAssertion'

$ dpkg-query -W nova-compute
nova-compute 2:12.0.3-0ubuntu1~cloud0

$ dpkg-query -W ceph
ceph 0.94.7-0ubuntu0.15.10.1~cloud0

$ dpkg-query -W libc-bin
libc-bin 2.19-0ubuntu6.9

Please let us know if you need any further information.

[Benjamin Kaehne (ben-kaehne)]

On all hosts observed; post the libc upgrade seemed to trigger after the following line in nova-compute:

2016-07-12 03:17:38.175 1227371 INFO nova.compute.resource_tracker [req-1c5222da-35ab-44b9-86bb-a9bd0ef88a85 - - - - -] Auditing locally available compute resources for node HOST.maas

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Peter Sabaini (peter-sabaini)]

Seen this on another node:

common/ceph_crypto.cc: In function 'void ceph::crypto::init(CephContext*)' thread 7f03da3b6740 time 2016-07-11 21:19:27.833484
common/ceph_crypto.cc: 73: FAILED assert(crypto_context != __null)
 ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)
 1: (()+0x11d2ab) [0x7f03703b72ab]
 2: (()+0x16ad60) [0x7f0370404d60]
 3: (()+0x1372e9) [0x7f03703d12e9]
 4: (()+0x1347b0) [0x7f03703ce7b0]
 5: (()+0x6704f) [0x7f037030104f]
 6: (ffi_call_unix64()+0x4c) [0x7f03d8093cce]
 7: (ffi_call()+0x308) [0x7f03d8093738]
 8: (_ctypes_callproc()+0x48e) [0x7f03d7ac05fe]
 9: (()+0x15f9e) [0x7f03d7ac1f9e]
 10: (PyObject_Call()+0x36) [0x505f96]
 11: (PyEval_EvalFrameEx()+0x1dfa) [0x49b07a]
 12: (PyEval_EvalFrameEx()+0xc72) [0x499ef2]
 13: (PyEval_EvalFrameEx()+0xc72) [0x499ef2]
 14: /usr/bin/python() [0x4a1c9a]
 15: (PyObject_Call()+0x36) [0x505f96]
 16: (PyEval_EvalFrameEx()+0x1dfa) [0x49b07a]
 17: /usr/bin/python() [0x4a1c9a]
 18: (PyEval_CallObjectWithKeywords()+0xf6) [0x4dca56]
 19: (()+0x392c) [0x7f03d909792c]
 20: (()+0x3276) [0x7f03d9097276]
 21: (()+0x3dc6) [0x7f03d9097dc6]
 22: (PyEval_EvalFrameEx()+0x3259) [0x49c4d9]
 23: /usr/bin/python() [0x4a1c9a]
 24: /usr/bin/python() [0x4dfe94]
 25: (PyObject_Call()+0x36) [0x505f96]
 26: /usr/bin/python() [0x4dddad]
 27: (PyEval_EvalFrameEx()+0x965) [0x499be5]
 28: (PyEval_EvalFrameEx()+0xc72) [0x499ef2]
 29: /usr/bin/python() [0x4a1c9a]
 30: /usr/bin/python() [0x4dfe94]
 31: (PyEval_CallObjectWithKeywords()+0x6b) [0x4dc9cb]
 32: (()+0x392c) [0x7f03d909792c]
 33: (()+0x3276) [0x7f03d9097276]
 34: /usr/bin/python() [0x5d9538]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
terminate called after throwing an instance of 'ceph::FailedAssertion'

# dpkg-query -W nova-compute ceph libc-bin
ceph 0.94.7-0ubuntu0.15.10.1~cloud0
libc-bin 2.19-0ubuntu6.9
nova-compute 2:12.0.3-0ubuntu1~cloud0