I think it's duplicate of another problem (#1517926) that was fixed in:
rpodolyaka@rpodolyaka-pc:~/src/nova$ git tag --contains e0647dd4b2ae9f5f6f908102d2ac447440622785
12.0.1
12.0.2
12.0.3
12.0.4
rpodolyaka@rpodolyaka-pc:~/src/nova$ git show e0647dd4b2ae9f5f6f908102d2ac447440622785
commit e0647dd4b2ae9f5f6f908102d2ac447440622785
Author: Roman Podoliaka <email address hidden>
Date: Thu Nov 19 16:00:01 2015 +0200
servicegroup: stop zombie service due to exception
If an exception is raised out of the _report_state call, we find that
the service no longer reports any updates to the database, so the
service is considered dead, thus creating a kind of zombie service.
I55417a5b91282c69432bb2ab64441c5cea474d31 seems to introduce a
regression, which leads to nova-* services marked as 'down', if an
error happens in a remote nova-conductor while processing a state
report: only Timeout errors are currently handled, but other errors
are possible, e.g. a DBError (wrapped with RemoteError on RPC
client side), if a DB temporarily goes away. This unhandled exception
will effectively break the state reporting thread - service will be
up again only after restart.
While the intention of I55417a5b91282c69432bb2ab64441c5cea474d31 was
to avoid cathing all the possible exceptions, but it looks like we must
do that to avoid creating a zombie.
The other part of that change was to ensure that during upgrade, we do
not spam the log server about MessagingTimeouts while the
nova-conductors are being restarted. This change ensures that still
happens.
Closes-Bug: #1517926
Change-Id: I44f118f82fbb811b790222face4c74d79795fe21
(cherry picked from commit 49b0d1741c674714fabf24d8409810064b953202)
and you seem to be using version 12.0.0.
Please try to update to the latest stable/liberty version of code and re-open the bug if it's still reproduced (I did not manage to reproduce it locally, presumably due to the fact that I use the version with the commit above ^ applied).
kaka
I think it's duplicate of another problem (#1517926) that was fixed in:
rpodolyaka@ rpodolyaka- pc:~/src/ nova$ git tag --contains e0647dd4b2ae9f5 f6f908102d2ac44 7440622785 rpodolyaka- pc:~/src/ nova$ git show e0647dd4b2ae9f5 f6f908102d2ac44 7440622785 f6f908102d2ac44 7440622785
12.0.1
12.0.2
12.0.3
12.0.4
rpodolyaka@
commit e0647dd4b2ae9f5
Author: Roman Podoliaka <email address hidden>
Date: Thu Nov 19 16:00:01 2015 +0200
servicegroup: stop zombie service due to exception
If an exception is raised out of the _report_state call, we find that
the service no longer reports any updates to the database, so the
service is considered dead, thus creating a kind of zombie service.
I55417a5b91 282c69432bb2ab6 4441c5cea474d31 seems to introduce a
regression, which leads to nova-* services marked as 'down', if an
error happens in a remote nova-conductor while processing a state
report: only Timeout errors are currently handled, but other errors
are possible, e.g. a DBError (wrapped with RemoteError on RPC
client side), if a DB temporarily goes away. This unhandled exception
will effectively break the state reporting thread - service will be
up again only after restart.
While the intention of I55417a5b91282c 69432bb2ab64441 c5cea474d31 was
to avoid cathing all the possible exceptions, but it looks like we must
do that to avoid creating a zombie.
The other part of that change was to ensure that during upgrade, we do
not spam the log server about MessagingTimeouts while the
nova-conductors are being restarted. This change ensures that still
happens.
Closes-Bug: #1517926
Change-Id: I44f118f82fbb81 1b790222face4c7 4d79795fe21 4fabf24d8409810 064b953202)
(cherry picked from commit 49b0d1741c67471
and you seem to be using version 12.0.0.
Please try to update to the latest stable/liberty version of code and re-open the bug if it's still reproduced (I did not manage to reproduce it locally, presumably due to the fact that I use the version with the commit above ^ applied).