MAAS

Sometimes it takes 25min to deploy a VM

Bug #2044417 reported by Jeffrey Chang
This bug report is a duplicate of:  Bug #2043997: Deployment calls time out. Edit Remove
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
New
Undecided
Unassigned

Bug Description

This happens when trying to request 2 machines from juju to create 2 additional controller nodes for HA.
test run : https://solutions.qa.canonical.com/testruns/c8721f0f-4fb9-4d3b-a316-4766ac9f0c77

juju status reported
'machine-status': {'current': 'allocating', 'message': 'failed to start machine 2 (unexpected: ServerError: 504 Gateway Time-out (<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx/1.18.0 (Ubuntu)</center>\r\n</body>\r\n</html>\r\n)), retrying in 10s (10 more attempts)', 'since': '18 Nov 2023 19:50:10Z'}

from /var/snap/maas/common/log/http/access.log, (erxfnq is juju-29-2-7 which is the expected instance here)
10.246.166.247 - - [18/Nov/2023:19:50:09 +0000] "POST /MAAS/api/2.0/machines/erxfnq/?op=deploy HTTP/1.1" 504 176 "-" "Go-http-client/1.1"

from maas.log
2023-11-18T19:50:09.907424+00:00 maas-lab1 maas.node: [info] juju-29-2-7: Releasing node
2023-11-18T19:50:09.907499+00:00 maas-lab1 maas.node: [info] juju-29-2-7: Status transition from DEPLOYING to RELEASING
2023-11-18T19:50:09.973735+00:00 maas-lab1 maas.node: [info] juju-29-2-7: Status transition from RELEASING to READY
2023-11-18T19:50:20.977769+00:00 maas-lab1 maas.node: [info] juju-29-2-7: Status transition from READY to ALLOCATED
2023-11-18T19:50:21.048565+00:00 maas-lab1 maas.node: [info] juju-29-2-7: allocated to user root
2023-11-18T19:51:00.114077+00:00 maas-lab1 maas.node: [warn] juju-29-2-7: Has a unknown BIOS boot method 'ipxe'; defaulting to 'pxe'.
2023-11-18T19:51:00.178784+00:00 maas-lab1 maas.node: [warn] juju-29-2-7: Has a unknown BIOS boot method 'ipxe'; defaulting to 'pxe'.
2023-11-18T19:51:00.289427+00:00 maas-lab1 maas.node: [info] juju-29-2-7: Status transition from ALLOCATED to DEPLOYING
2023-11-18T19:51:38.783871+00:00 maas-lab1 maas.power: [info] Changing power state (on) of node: juju-29-2-7 (erxfnq)
2023-11-18T19:51:43.439604+00:00 maas-lab1 maas.power: [info] Changed power state (on) of node: juju-29-2-7 (erxfnq)
2023-11-18T19:52:17.841548+00:00 maas-lab1 maas.node: [warn] juju-29-2-7: Has a unknown BIOS boot method 'ipxe'; defaulting to 'pxe'.
2023-11-18T19:52:17.852922+00:00 maas-lab1 maas.node: [warn] juju-29-2-7: Has a unknown BIOS boot method 'ipxe'; defaulting to 'pxe'.
2023-11-18T20:10:15.766946+00:00 maas-lab1 maas.node: [info] juju-29-2-7: Status transition from DEPLOYING to DEPLOYED
2023-11-18T20:20:53.963332+00:00 maas-lab1 maas.node: [info] juju-29-2-7: Releasing node
2023-11-18T20:20:53.981843+00:00 maas-lab1 maas.node: [info] juju-29-2-7: Status transition from DEPLOYED to RELEASING
2023-11-18T20:20:54.197325+00:00 maas-lab1 maas.power: [info] Changing power state (off) of node: juju-29-2-7 (erxfnq)

Since the SOS report would be hundreds MB(this is a long running MAAS), please ask SolQA if you want to access to the env.

Revision history for this message
Jeffrey Chang (modern911) wrote :
Download full text (4.1 KiB)

Another case is for juju-60-1-16(nmqt8w) around 2023-11-18T18:10
for testrun https://solutions.qa.canonical.com/testruns/c8721f0f-4fb9-4d3b-a316-4766ac9f0c77

2023-11-18T18:10:53.527792+00:00 maas-lab1 maas.node: [info] juju-60-1-16: Status transition from READY to ALLOCATED
2023-11-18T18:10:53.573497+00:00 maas-lab1 maas.node: [info] juju-60-1-16: allocated to user root
2023-11-18T18:11:41.701395+00:00 maas-lab1 maas.node: [warn] juju-60-1-16: Has a unknown BIOS boot method 'ipxe'; defaulting to 'pxe'.
2023-11-18T18:11:41.778445+00:00 maas-lab1 maas.node: [warn] juju-60-1-16: Has a unknown BIOS boot method 'ipxe'; defaulting to 'pxe'.
2023-11-18T18:11:41.991441+00:00 maas-lab1 maas.node: [info] juju-60-1-16: Status transition from ALLOCATED to DEPLOYING
2023-11-18T18:27:20.436292+00:00 maas-lab1 maas.node: [info] juju-60-1-16: Releasing node
2023-11-18T18:27:20.436386+00:00 maas-lab1 maas.node: [info] juju-60-1-16: Status transition from DEPLOYING to RELEASING
2023-11-18T18:27:20.555816+00:00 maas-lab1 maas.node: [info] juju-60-1-16: Status transition from RELEASING to READY
2023-11-18T18:29:25.350197+00:00 maas-lab1 maas.node: [info] juju-60-1-16: Status transition from READY to ALLOCATED
2023-11-18T18:29:25.376783+00:00 maas-lab1 maas.node: [info] juju-60-1-16: allocated to user root
2023-11-18T18:30:29.436991+00:00 maas-lab1 maas.node: [warn] juju-60-1-16: Has a unknown BIOS boot method 'ipxe'; defaulting to 'pxe'.
2023-11-18T18:30:29.477340+00:00 maas-lab1 maas.node: [warn] juju-60-1-16: Has a unknown BIOS boot method 'ipxe'; defaulting to 'pxe'.
2023-11-18T18:30:29.793463+00:00 maas-lab1 maas.node: [info] juju-60-1-16: Status transition from ALLOCATED to DEPLOYING
2023-11-18T18:31:19.507404+00:00 maas-lab1 maas.power: [info] Changing power state (on) of node: juju-60-1-16 (nmqt8w)
2023-11-18T18:31:24.024045+00:00 maas-lab1 maas.power: [info] Changed power state (on) of node: juju-60-1-16 (nmqt8w)
2023-11-18T18:32:04.195518+00:00 maas-lab1 maas.node: [warn] juju-60-1-16: Has a unknown BIOS boot method 'ipxe'; defaulting to 'pxe'.
2023-11-18T18:32:04.239961+00:00 maas-lab1 maas.node: [warn] juju-60-1-16: Has a unknown BIOS boot method 'ipxe'; defaulting to 'pxe'.
2023-11-18T18:35:19.047780+00:00 maas-lab1 maas.node: [info] juju-60-1-16: Status transition from DEPLOYING to DEPLOYED
2023-11-18T18:48:22.769006+00:00 maas-lab1 maas.node: [info] juju-60-1-16: Releasing node
2023-11-18T18:48:22.783730+00:00 maas-lab1 maas.node: [info] juju-60-1-16: Status transition from DEPLOYED to RELEASING
2023-11-18T18:48:23.044229+00:00 maas-lab1 maas.power: [info] Changing power state (off) of node: juju-60-1-16 (nmqt8w)
2023-11-18T18:48:27.390327+00:00 maas-lab1 maas.node: [info] juju-60-1-16: Status transition from RELEASING to READY
2023-11-18T18:48:27.428559+00:00 maas-lab1 maas.power: [info] Changed power state (off) of node: juju-60-1-16 (nmqt8w)

normally, this should takes only <5min
2023-11-20T15:30:59.543756+00:00 maas-lab1 maas.node: [info] juju-60-1-16: Status transition from READY to ALLOCATED
2023-11-20T15:30:59.561979+00:00 maas-lab1 maas.node: [info] juju-60-1-16: allocated to user root
2023-11-20T15:31:15.462773+00:00 maas-lab1 maas...

Read more...

summary: - unexpected: ServerError: 504 Gateway Time-out when requesting machine by
- juju
+ Sometimes it takes 25min to deploy a VM
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.