[2.0 RC2] Multiple failures to release nodes
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Fix Released
|
High
|
Lee Trager | ||
2.0 |
Fix Released
|
High
|
Lee Trager | ||
2.1 |
Fix Released
|
High
|
Lee Trager |
Bug Description
I am seeing a number of servers that failed to release. These are automated builds not manual deployments. There were 4 failed nodes on one of the maas servers:
ubuntu@
Jul 14 20:56:11 maas2-production maas.node: [INFO] tucker: Status transition from RELEASING to FAILED_RELEASING
Jul 15 01:56:50 maas2-production maas.node: [INFO] orangebox10: Status transition from RELEASING to FAILED_RELEASING
ubuntu@
Jul 15 06:31:27 maas2-production maas.node: [INFO] orangebox9: Status transition from RELEASING to FAILED_RELEASING
After seeing these, I tried releasing the nodes manually through the UI and that worked.
ubuntu@
Jul 15 06:31:27 maas2-production maas.node: [INFO] orangebox9: Status transition from RELEASING to FAILED_RELEASING
Jul 15 18:22:55 maas2-production maas.node: [INFO] orangebox3: Status transition from FAILED_RELEASING to RELEASING
Jul 15 18:22:55 maas2-production maas.node: [INFO] orangebox9: Status transition from FAILED_RELEASING to RELEASING
Jul 15 18:22:55 maas2-production maas.node: [INFO] orangebox10: Status transition from FAILED_RELEASING to RELEASING
Jul 15 18:22:55 maas2-production maas.node: [INFO] orangebox6: Status transition from FAILED_RELEASING to RELEASING
ubuntu@
but then I saw another 4 failures on another maas server which is also on RC2. I left these in the failed state, but the node manager which polls the server states may try to release them.
ubuntu@
Jul 15 11:48:27 maas2-integration maas.node: [INFO] prunus: Status transition from RELEASING to FAILED_RELEASING
Jul 15 11:48:27 maas2-integration maas.node: [INFO] kobusch: Status transition from RELEASING to FAILED_RELEASING
Jul 15 19:35:17 maas2-integration maas.node: [INFO] hayward-54: Status transition from RELEASING to FAILED_RELEASING
Jul 15 19:35:22 maas2-integration maas.node: [INFO] hayward-63: Status transition from RELEASING to FAILED_RELEASING
ubuntu@
Desired=
| Status=
|/ Err?=(none)
||/ Name Version Architecture Description
+++-===
ii maas 2.0.0~rc2+
ii maas-cli 2.0.0~rc2+
un maas-cluster-
ii maas-common 2.0.0~rc2+
ii maas-dhcp 2.0.0~rc2+
ii maas-dns 2.0.0~rc2+
ii maas-proxy 2.0.0~rc2+
ii maas-rack-
ii maas-region-api 2.0.0~rc2+
ii maas-region-
un maas-region-
un python-django-maas <none> <none> (no description available)
un python-maas-client <none> <none> (no description available)
un python-
ii python3-django-maas 2.0.0~rc2+
ii python3-maas-client 2.0.0~rc2+
ii python3-
I am attaching the logs for both maas servers (logs1.tar.gz for first one and logs2.tar.gz for second server).
Related branches
- Mike Pontillo (community): Approve
-
Diff: 47 lines (+18/-0)2 files modifiedsrc/maasserver/models/node.py (+9/-0)
src/maasserver/models/tests/test_node.py (+9/-0)
- Lee Trager (community): Approve
-
Diff: 47 lines (+18/-0)2 files modifiedsrc/maasserver/models/node.py (+9/-0)
src/maasserver/models/tests/test_node.py (+9/-0)
Changed in maas: | |
status: | New → Triaged |
Hi Larry,
Can you please provide node event logs as well?