| 2016-07-15 20:41:55 |
Larry Michel |
bug |
|
|
added bug |
| 2016-07-15 20:41:55 |
Larry Michel |
attachment added |
|
logs.tar.gz https://bugs.launchpad.net/bugs/1603563/+attachment/4701746/+files/logs.tar.gz |
|
| 2016-07-15 20:45:52 |
Larry Michel |
description |
I am seeing a number of servers that failed to release. These are automated builds not manual deployments. There were 4 nodes on maas server:
ubuntu@maas2-production:/var/log/maas$ grep -i "failed_rele" maas.log
Jul 15 06:31:27 maas2-production maas.node: [INFO] orangebox9: Status transition from RELEASING to FAILED_RELEASING
Jul 15 18:22:55 maas2-production maas.node: [INFO] orangebox3: Status transition from FAILED_RELEASING to RELEASING
Jul 15 18:22:55 maas2-production maas.node: [INFO] orangebox9: Status transition from FAILED_RELEASING to RELEASING
Jul 15 18:22:55 maas2-production maas.node: [INFO] orangebox10: Status transition from FAILED_RELEASING to RELEASING
Jul 15 18:22:55 maas2-production maas.node: [INFO] orangebox6: Status transition from FAILED_RELEASING to RELEASING
ubuntu@maas2-production:/var/log/maas$
After seeing these, I tried releasing them manually and that worked.
but I saw another 4 failures on another maas server which is also on RC2. I left these in the failed state, but the node manager which polls the server states may try to release them.
ubuntu@maas2-integration:/var/log/maas$ grep -i "failed_rele" maas.log
Jul 15 11:48:27 maas2-integration maas.node: [INFO] prunus: Status transition from RELEASING to FAILED_RELEASING
Jul 15 11:48:27 maas2-integration maas.node: [INFO] kobusch: Status transition from RELEASING to FAILED_RELEASING
Jul 15 19:35:17 maas2-integration maas.node: [INFO] hayward-54: Status transition from RELEASING to FAILED_RELEASING
Jul 15 19:35:22 maas2-integration maas.node: [INFO] hayward-63: Status transition from RELEASING to FAILED_RELEASING
ubuntu@maas2-production:~$ dpkg -l '*maas*'|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-==================================-============-=================================================
ii maas 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all "Metal as a Service" is a physical cloud and IPAM
ii maas-cli 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS client and command-line interface
un maas-cluster-controller <none> <none> (no description available)
ii maas-common 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS server common files
ii maas-dhcp 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS DHCP server
ii maas-dns 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS DNS server
ii maas-proxy 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS Caching Proxy
ii maas-rack-controller 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all Rack Controller for MAAS
ii maas-region-api 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all Region controller API service for MAAS
ii maas-region-controller 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all Region Controller for MAAS
un maas-region-controller-min <none> <none> (no description available)
un python-django-maas <none> <none> (no description available)
un python-maas-client <none> <none> (no description available)
un python-maas-provisioningserver <none> <none> (no description available)
ii python3-django-maas 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS server Django web framework (Python 3)
ii python3-maas-client 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS python API client (Python 3)
ii python3-maas-provisioningserver 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS server provisioning libraries (Python 3)
I am attaching the logs for both maas servers (logs1.tar.gz for first one and logs2.tar.gz for second server). |
I am seeing a number of servers that failed to release. These are automated builds not manual deployments. There were 4 failed nodes on one of the maas servers:
ubuntu@maas2-production:/var/log/maas$ grep " to FAILED_RELEA" maas.log.1
Jul 14 20:56:11 maas2-production maas.node: [INFO] tucker: Status transition from RELEASING to FAILED_RELEASING
Jul 15 01:56:50 maas2-production maas.node: [INFO] orangebox10: Status transition from RELEASING to FAILED_RELEASING
ubuntu@maas2-production:/var/log/maas$ grep " to FAILED_RELEA" maas.log
Jul 15 06:31:27 maas2-production maas.node: [INFO] orangebox9: Status transition from RELEASING to FAILED_RELEASING
After seeing these, I tried releasing the nodes manually through the UI and that worked.
ubuntu@maas2-production:/var/log/maas$ grep -i "failed_rele" maas.log
Jul 15 06:31:27 maas2-production maas.node: [INFO] orangebox9: Status transition from RELEASING to FAILED_RELEASING
Jul 15 18:22:55 maas2-production maas.node: [INFO] orangebox3: Status transition from FAILED_RELEASING to RELEASING
Jul 15 18:22:55 maas2-production maas.node: [INFO] orangebox9: Status transition from FAILED_RELEASING to RELEASING
Jul 15 18:22:55 maas2-production maas.node: [INFO] orangebox10: Status transition from FAILED_RELEASING to RELEASING
Jul 15 18:22:55 maas2-production maas.node: [INFO] orangebox6: Status transition from FAILED_RELEASING to RELEASING
ubuntu@maas2-production:/var/log/maas$
but then I saw another 4 failures on another maas server which is also on RC2. I left these in the failed state, but the node manager which polls the server states may try to release them.
ubuntu@maas2-integration:/var/log/maas$ grep -i "failed_rele" maas.log
Jul 15 11:48:27 maas2-integration maas.node: [INFO] prunus: Status transition from RELEASING to FAILED_RELEASING
Jul 15 11:48:27 maas2-integration maas.node: [INFO] kobusch: Status transition from RELEASING to FAILED_RELEASING
Jul 15 19:35:17 maas2-integration maas.node: [INFO] hayward-54: Status transition from RELEASING to FAILED_RELEASING
Jul 15 19:35:22 maas2-integration maas.node: [INFO] hayward-63: Status transition from RELEASING to FAILED_RELEASING
ubuntu@maas2-production:~$ dpkg -l '*maas*'|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-==================================-============-=================================================
ii maas 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all "Metal as a Service" is a physical cloud and IPAM
ii maas-cli 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS client and command-line interface
un maas-cluster-controller <none> <none> (no description available)
ii maas-common 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS server common files
ii maas-dhcp 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS DHCP server
ii maas-dns 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS DNS server
ii maas-proxy 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS Caching Proxy
ii maas-rack-controller 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all Rack Controller for MAAS
ii maas-region-api 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all Region controller API service for MAAS
ii maas-region-controller 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all Region Controller for MAAS
un maas-region-controller-min <none> <none> (no description available)
un python-django-maas <none> <none> (no description available)
un python-maas-client <none> <none> (no description available)
un python-maas-provisioningserver <none> <none> (no description available)
ii python3-django-maas 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS server Django web framework (Python 3)
ii python3-maas-client 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS python API client (Python 3)
ii python3-maas-provisioningserver 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS server provisioning libraries (Python 3)
I am attaching the logs for both maas servers (logs1.tar.gz for first one and logs2.tar.gz for second server). |
|
| 2016-07-15 21:35:05 |
Andres Rodriguez |
maas: status |
New |
Incomplete |
|
| 2016-07-20 12:25:49 |
Larry Michel |
attachment added |
|
events-logs.tar.gz https://bugs.launchpad.net/maas/+bug/1603563/+attachment/4704035/+files/events-logs.tar.gz |
|
| 2016-07-20 12:25:57 |
Larry Michel |
maas: status |
Incomplete |
New |
|
| 2016-07-20 12:51:20 |
Andres Rodriguez |
maas: importance |
Undecided |
High |
|
| 2016-07-20 12:51:24 |
Andres Rodriguez |
maas: milestone |
|
2.0.0 |
|
| 2016-07-20 12:51:32 |
Andres Rodriguez |
nominated for series |
|
maas/2.0 |
|
| 2016-07-20 12:51:32 |
Andres Rodriguez |
bug task added |
|
maas/2.0 |
|
| 2016-07-20 12:51:32 |
Andres Rodriguez |
nominated for series |
|
maas/trunk |
|
| 2016-07-20 12:51:32 |
Andres Rodriguez |
bug task added |
|
maas/trunk |
|
| 2016-07-20 12:51:37 |
Andres Rodriguez |
maas/2.0: milestone |
|
2.0.0 |
|
| 2016-07-20 12:51:40 |
Andres Rodriguez |
maas/trunk: milestone |
2.0.0 |
2.1.0 |
|
| 2016-07-20 13:09:59 |
Larry Michel |
attachment added |
|
logs-Jul17.tar.gz https://bugs.launchpad.net/maas/+bug/1603563/+attachment/4704047/+files/logs-Jul17.tar.gz |
|
| 2016-07-21 20:21:21 |
Newell Jensen |
maas/2.0: assignee |
|
Newell Jensen (newell-jensen) |
|
| 2016-07-21 20:21:23 |
Newell Jensen |
maas/trunk: assignee |
|
Newell Jensen (newell-jensen) |
|
| 2016-07-21 20:21:26 |
Newell Jensen |
maas/2.0: importance |
Undecided |
High |
|
| 2016-07-22 16:28:47 |
Newell Jensen |
maas/trunk: status |
New |
Incomplete |
|
| 2016-07-22 16:28:49 |
Newell Jensen |
maas/2.0: status |
New |
Incomplete |
|
| 2016-08-15 13:08:05 |
Larry Michel |
attachment added |
|
Screenshot from 2016-08-15 08-04-03.png https://bugs.launchpad.net/maas/+bug/1603563/+attachment/4721236/+files/Screenshot%20from%202016-08-15%2008-04-03.png |
|
| 2016-08-15 13:08:11 |
Larry Michel |
maas/2.0: status |
Incomplete |
New |
|
| 2016-08-15 13:08:14 |
Larry Michel |
maas/trunk: status |
Incomplete |
New |
|
| 2016-08-23 11:37:46 |
Andres Rodriguez |
maas/2.0: status |
New |
Triaged |
|
| 2016-08-23 11:37:48 |
Andres Rodriguez |
maas/trunk: status |
New |
Triaged |
|
| 2016-08-23 14:08:19 |
Andres Rodriguez |
maas/2.0: status |
Triaged |
Incomplete |
|
| 2016-08-23 14:08:21 |
Andres Rodriguez |
maas/trunk: status |
Triaged |
Incomplete |
|
| 2016-08-23 14:08:24 |
Andres Rodriguez |
maas/trunk: milestone |
2.0.1 |
|
|
| 2016-08-23 14:08:29 |
Andres Rodriguez |
maas/2.0: milestone |
2.0.0 |
|
|
| 2016-08-24 01:18:16 |
Larry Michel |
attachment added |
|
Screenshot from 2016-08-23 20-09-43.png https://bugs.launchpad.net/maas/+bug/1603563/+attachment/4726695/+files/Screenshot%20from%202016-08-23%2020-09-43.png |
|
| 2016-08-24 01:18:38 |
Larry Michel |
maas/2.0: status |
Incomplete |
New |
|
| 2016-08-24 01:18:41 |
Larry Michel |
maas/trunk: status |
Incomplete |
New |
|
| 2016-08-24 01:24:22 |
Larry Michel |
attachment added |
|
logs.tar.gz https://bugs.launchpad.net/maas/+bug/1603563/+attachment/4726698/+files/logs.tar.gz |
|
| 2016-08-24 13:25:00 |
Andres Rodriguez |
maas/2.0: milestone |
|
2.1.0 |
|
| 2016-08-24 13:25:02 |
Andres Rodriguez |
maas/2.0: milestone |
2.1.0 |
2.0.1 |
|
| 2016-09-26 13:55:36 |
Gavin Panella |
maas: status |
New |
Triaged |
|
| 2016-09-26 13:56:01 |
Gavin Panella |
maas/2.0: status |
New |
Triaged |
|
| 2016-10-24 19:19:38 |
Andres Rodriguez |
nominated for series |
|
maas/2.1 |
|
| 2016-10-24 19:19:38 |
Andres Rodriguez |
bug task added |
|
maas/2.1 |
|
| 2016-10-24 20:13:47 |
David Britton |
attachment added |
|
maas-logs.tar.gz https://bugs.launchpad.net/maas/+bug/1603563/+attachment/4766644/+files/maas-logs.tar.gz |
|
| 2016-10-24 23:55:33 |
David Britton |
attachment added |
|
maas-logs.tar.gz https://bugs.launchpad.net/maas/+bug/1603563/+attachment/4766724/+files/maas-logs.tar.gz |
|
| 2016-10-24 23:56:55 |
David Britton |
attachment added |
|
release-node-failure.png https://bugs.launchpad.net/maas/+bug/1603563/+attachment/4766725/+files/release-node-failure.png |
|
| 2016-10-25 20:09:59 |
Launchpad Janitor |
branch linked |
|
lp:~ltrager/maas/lp1603563 |
|
| 2016-10-26 19:32:57 |
MAAS Lander |
maas/trunk: status |
Triaged |
Fix Committed |
|
| 2016-10-26 20:31:01 |
Lee Trager |
maas/trunk: assignee |
Newell Jensen (newell-jensen) |
Lee Trager (ltrager) |
|
| 2016-10-26 20:31:04 |
Lee Trager |
maas/2.1: assignee |
|
Lee Trager (ltrager) |
|
| 2016-10-26 20:31:07 |
Lee Trager |
maas/2.0: assignee |
Newell Jensen (newell-jensen) |
Lee Trager (ltrager) |
|
| 2016-10-26 20:57:09 |
Andres Rodriguez |
maas/2.1: milestone |
|
2.1.1 |
|
| 2016-10-26 20:57:15 |
Andres Rodriguez |
maas/2.1: status |
New |
Fix Committed |
|
| 2016-10-26 23:48:32 |
Lee Trager |
maas/2.1: importance |
Undecided |
High |
|
| 2016-10-27 00:35:26 |
Launchpad Janitor |
branch linked |
|
lp:~ltrager/maas/lp1603563_2.0 |
|
| 2016-10-27 01:19:55 |
MAAS Lander |
maas/2.0: status |
Triaged |
Fix Committed |
|
| 2016-12-08 08:25:22 |
LaMont Jones |
maas/trunk: milestone |
|
2.2.0 |
|
| 2016-12-08 08:55:04 |
Lee Trager |
maas/2.0: status |
Fix Committed |
Fix Released |
|
| 2016-12-08 08:55:06 |
Lee Trager |
maas/2.1: status |
Fix Committed |
Fix Released |
|
| 2016-12-08 08:55:09 |
Lee Trager |
maas/trunk: status |
Fix Committed |
Fix Released |
|
