nova-compute process killed after libc-bin upgrade
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
nova-compute (Ubuntu) |
Confirmed
|
Undecided
|
Unassigned |
Bug Description
nova-compute process across 2 Openstack deployments died around the same time as the recent libc-bin upgrade:
2016-07-12 00:59:26 status half-configured libc-bin:amd64 2.19-0ubuntu6.9
2016-07-12 00:59:26 status installed libc-bin:amd64 2.19-0ubuntu6.9
Digging into the logs showed it was possibly related to ceph:
common/
common/
ceph version 0.94.6 (e832001feaf8c1
1: (()+0x11d2ab) [0x7fbb693b92ab]
2: (()+0x16ad60) [0x7fbb69406d60]
3: (()+0x1372e9) [0x7fbb693d32e9]
4: (()+0x1347b0) [0x7fbb693d07b0]
5: (()+0x6704f) [0x7fbb6930304f]
6: (ffi_call_
7: (ffi_call()+0x308) [0x7fbbd9447738]
8: (_ctypes_
9: (()+0x15f9e) [0x7fbbd8e75f9e]
10: (PyObject_
11: (PyEval_
12: (PyEval_
13: (PyEval_
14: /usr/bin/python() [0x4a1c9a]
15: (PyObject_
16: (PyEval_
17: /usr/bin/python() [0x4a1c9a]
18: (PyEval_
19: (()+0x392c) [0x7fbbda44b92c]
20: (()+0x3276) [0x7fbbda44b276]
21: (()+0x3dc6) [0x7fbbda44bdc6]
22: (PyEval_
23: /usr/bin/python() [0x4a1c9a]
24: /usr/bin/python() [0x4dfe94]
25: (PyObject_
26: /usr/bin/python() [0x4dddad]
27: (PyEval_
28: (PyEval_
29: /usr/bin/python() [0x4a1c9a]
30: /usr/bin/python() [0x4dfe94]
31: (PyEval_
32: (()+0x392c) [0x7fbbda44b92c]
33: (()+0x3276) [0x7fbbda44b276]
34: /usr/bin/python() [0x5d9538]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
terminate called after throwing an instance of 'ceph::
$ dpkg-query -W nova-compute
nova-compute 2:12.0.
$ dpkg-query -W ceph
ceph 0.94.7-
$ dpkg-query -W libc-bin
libc-bin 2.19-0ubuntu6.9
Please let us know if you need any further information.
On all hosts observed; post the libc upgrade seemed to trigger after the following line in nova-compute:
2016-07-12 03:17:38.175 1227371 INFO nova.compute. resource_ tracker [req-1c5222da- 35ab-44b9- 86bb-a9bd0ef88a 85 - - - - -] Auditing locally available compute resources for node HOST.maas