Ceilometer API service stuck - Address already in use
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Mirantis OpenStack |
Won't Fix
|
Low
|
MOS Maintenance | ||
9.x |
Fix Released
|
High
|
Ilya Tyaptin |
Bug Description
MOS 9.2
During the tests, the communication with the ceilometer-api on one of the controllers suddenly fails.
From the ceilometer-api.log:
2017-03-
2017-03-
2017-03-
Workaround: restart the service.
Haven't found a way to reproduce that.
-------
Additional information:
To probe the status of the Ceilometer services we manually created a Curl request to address locally to every ceilometer instance on the CICs.
[From any of the CIC]
curl -g -i -X 'GET' 'http://<CIC_IP_
Result: request goes through all the CICs (Result: 200 OK) except CIC-1 where it gets a time out.
>>>>> ceilomter-api service status <<<<<<
We checked the status of the service with the following results
>>>>> strace
root@cic-1:~# strace -tt -T -p 5596
Process 5596 attached
19:48:39.985419 wait4(0, <- the ceilomter-api master got stuck here
>>>>> GDB
>> CIC-1 !Faulty!
(gdb) bt
#0 0x00007f2feb0dced9 in waitpid () from /lib/x86_
#1 0x000000000041d95a in ?? ()
#2 0x000000000049968d in PyEval_EvalFrameEx ()
#3 0x0000000000499ef2 in PyEval_EvalFrameEx ()
#4 0x0000000000499ef2 in PyEval_EvalFrameEx ()
#5 0x0000000000499ef2 in PyEval_EvalFrameEx ()
#6 0x00000000004a1c9a in ?? ()
#7 0x00000000004dfe94 in ?? ()
#8 0x0000000000499be5 in PyEval_EvalFrameEx ()
#9 0x00000000004a090c in PyEval_EvalCodeEx ()
#10 0x000000000049ab45 in PyEval_EvalFrameEx ()
#11 0x00000000004a090c in PyEval_EvalCodeEx ()
#12 0x000000000049ab45 in PyEval_EvalFrameEx ()
#13 0x00000000004a090c in PyEval_EvalCodeEx ()
#14 0x0000000000499a52 in PyEval_EvalFrameEx ()
#15 0x0000000000499ef2 in PyEval_EvalFrameEx ()
#16 0x0000000000499ef2 in PyEval_EvalFrameEx ()
#17 0x00000000004a1634 in ?? ()
#18 0x000000000044e4a5 in PyRun_FileExFlags ()
#19 0x000000000044ec9f in PyRun_SimpleFil
#20 0x000000000044f904 in Py_Main ()
#21 0x00007f2fead29f45 in __libc_start_main () from /lib/x86_
#22 0x0000000000578c4e in _start ()
>> CIC-2 !working fine!
(gdb) bt
#0 0x00007f7d3376dc53 in select () from /lib/x86_
#1 0x000000000047fbbd in ?? ()
#2 0x000000000049c4d9 in PyEval_EvalFrameEx ()
#3 0x00000000004a090c in PyEval_EvalCodeEx ()
#4 0x000000000049ab45 in PyEval_EvalFrameEx ()
#5 0x00000000004a1c9a in ?? ()
#6 0x00000000004dfe94 in ?? ()
#7 0x0000000000499be5 in PyEval_EvalFrameEx ()
#8 0x00000000004a090c in PyEval_EvalCodeEx ()
#9 0x000000000049ab45 in PyEval_EvalFrameEx ()
#10 0x00000000004a090c in PyEval_EvalCodeEx ()
#11 0x000000000049ab45 in PyEval_EvalFrameEx ()
#12 0x00000000004a090c in PyEval_EvalCodeEx ()
#13 0x0000000000499a52 in PyEval_EvalFrameEx ()
#14 0x0000000000499ef2 in PyEval_EvalFrameEx ()
#15 0x0000000000499ef2 in PyEval_EvalFrameEx ()
#16 0x00000000004a1634 in ?? ()
#17 0x000000000044e4a5 in PyRun_FileExFlags ()
#18 0x000000000044ec9f in PyRun_SimpleFil
#19 0x000000000044f904 in Py_Main ()
#20 0x00007f7d3369df45 in __libc_start_main () from /lib/x86_
#21 0x0000000000578c4e in _start ()
>>>>> /proc/<
>> CIC-1 !Faulty!
[<ffffffff8107f
[<ffffffff81080
[<ffffffff817fa
[<fffffffffffff
>> CIC-2 !Working fine!
[<ffffffff81211
[<ffffffff81212
[<ffffffff81212
[<ffffffff81212
[<ffffffff817fa
[<fffffffffffff
The root cause seems to be related to a deadlock in the ceilometer-api
-------
Also, there is a bug that could be potentially related:
https:/
Changed in mos: | |
importance: | Undecided → High |
description: | updated |
summary: |
- Ceilometer API service doesn't start - Address already in use + Ceilometer API service can't start - Address already in use |
summary: |
- Ceilometer API service can't start - Address already in use + Ceilometer API service stuck - Address already in use |
Changed in mos: | |
assignee: | nobody → MOS Ceilometer (mos-ceilometer) |
status: | New → Confirmed |
milestone: | none → 10.0 |
tags: | added: area-ceilometer |
Changed in mos: | |
assignee: | MOS Ceilometer (mos-ceilometer) → Ilya Tyaptin (ityaptin) |
Changed in mos: | |
assignee: | Ilya Tyaptin (ityaptin) → MOS Maintenance (mos-maintenance) |
sla1 for 9.0-updates