MAAS

juju destroy-environment >=256 nodes fails

Bug #1316272 reported by James Page
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
MAAS
Invalid
Undecided
Unassigned
juju-core
Fix Released
High
Andrew Wilkins
juju-core 1.19.3
juju-core (Ubuntu)
Fix Released
Medium
Unassigned
maas (Ubuntu)
Invalid
Undecided
Unassigned

Bug Description

juju destroy-environment in a large maas deployment (~256 servers) fails with:

ERROR destroying environment: gomaasapi: got error back from server: 414 Request-URI Too Long (<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>414 Request-URI Too Long</title>
</head><body>
<h1>Request-URI Too Long</h1>
<p>The requested URL's length exceeds the capacity
limit for this server.<br />
</p>
<hr>
<address>Apache/2.4.7 (Ubuntu) Server at batMASS.amd.com Port 80</address>
</body></html>
)

Using the --force option worked around this.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: maas 1.5+bzr2252-0ubuntu1
ProcVersionSignature: Ubuntu 3.13.0-24.46-generic 3.13.9
Uname: Linux 3.13.0-24-generic x86_64
ApportVersion: 2.14.1-0ubuntu3
Architecture: amd64
Date: Mon May 5 12:04:35 2014
InstallationDate: Installed on 2014-04-18 (17 days ago)
InstallationMedia: Ubuntu-Server 14.04 LTS "Trusty Tahr" - Release amd64 (20140416.2)
PackageArchitecture: all
ProcEnviron:
 TERM=screen
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: maas
UpgradeStatus: No upgrade log present (probably fresh install)

Related branches

lp:~axwalk/juju-core/lp1260171-stopinstances-ids
Juju Engineering: Pending requested
Diff: 1305 lines (+232/-257)
38 files modified
container/interface.go (+7/-3)
container/kvm/kvm.go (+2/-2)
container/kvm/kvm_test.go (+1/-1)
container/kvm/live_test.go (+1/-1)
container/lxc/lxc.go (+2/-2)
container/lxc/lxc_test.go (+5/-5)
dependencies.tsv (+1/-1)
environs/broker.go (+3/-2)
environs/jujutest/livetests.go (+6/-6)
environs/jujutest/tests.go (+1/-1)
provider/azure/environ.go (+17/-16)
provider/azure/environ_test.go (+11/-12)
provider/common/bootstrap.go (+1/-1)
provider/common/bootstrap_test.go (+7/-7)
provider/common/destroy.go (+6/-1)
provider/common/destroy_test.go (+11/-11)
provider/common/mock_test.go (+3/-3)
provider/dummy/environs.go (+7/-7)
provider/ec2/ec2.go (+1/-5)
provider/ec2/live_test.go (+5/-5)
provider/joyent/environ_instance.go (+39/-6)
provider/joyent/instance.go (+0/-41)
provider/joyent/local_test.go (+3/-3)
provider/local/environ.go (+5/-5)
provider/maas/environ.go (+11/-25)
provider/maas/environ_whitebox_test.go (+17/-13)
provider/maas/util.go (+3/-3)
provider/maas/util_test.go (+1/-1)
provider/manual/environ.go (+1/-1)
provider/openstack/local_test.go (+8/-8)
provider/openstack/provider.go (+23/-19)
state/apiserver/client/destroy.go (+1/-22)
worker/provisioner/kvm-broker.go (+4/-4)
worker/provisioner/kvm-broker_test.go (+3/-3)
worker/provisioner/lxc-broker.go (+4/-4)
worker/provisioner/lxc-broker_test.go (+3/-3)
worker/provisioner/provisioner_task.go (+6/-2)
worker/provisioner/provisioner_test.go (+2/-2)
Revision history for this message
James Page (james-page) wrote :
Curtis Hovey (sinzui)
Changed in juju-core:
status: New → Triaged
importance: Undecided → High
milestone: none → 1.19.2
tags: added: destroy-environment maas-provider
Revision history for this message
Julian Edwards (julian-edwards) wrote :

It looks like that error came from Apache, so I'm going to invalidate the maas task. If you think I'm wrong, please open it again and tell me why, I won't be offended.

Changed in maas:
status: New → Invalid
Revision history for this message
John A Meinel (jameinel) wrote : Re: [Bug 1316272] Re: juju destroy-environment >=256 nodes fails

So this sounds like we are putting all the machine IDs into a single URL
and we end up running out of URL space (quick google says that the default
max length is 2000 characters).

It sounds like we either need to send the request in batches, or POST the
IDs rather than put them in the URL itself.

John
=:->

On Tue, May 6, 2014 at 4:07 AM, Julian Edwards
<email address hidden>wrote:

> It looks like that error came from Apache, so I'm going to invalidate
> the maas task. If you think I'm wrong, please open it again and tell me
> why, I won't be offended.
>
> ** Changed in: maas
> Status: New => Invalid
>
> --
> You received this bug notification because you are subscribed to juju-
> core.
> https://bugs.launchpad.net/bugs/1316272
>
> Title:
> juju destroy-environment >=256 nodes fails
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1316272/+subscriptions
>

Revision history for this message
Julian Edwards (julian-edwards) wrote :

On 06/05/14 14:03, John A Meinel wrote:
> It sounds like we either need to send the request in batches, or POST the
> IDs rather than put them in the URL itself.

POST should work and is a lot easier to implement, I hope, if the Go
stdlib supports a simple switch...

Curtis Hovey (sinzui)
Changed in juju-core:
milestone: 1.19.2 → 1.19.3
Revision history for this message
James Page (james-page) wrote :

worked around with --force for the time being

Changed in maas (Ubuntu):
status: New → Invalid
tags: added: sm15k
Revision history for this message
Andrew Wilkins (axwalk) wrote :

We already do POST when stopping instances. The problem is likely that we first have to map instance IDs to instances so we can stop them (this is dumb and needs to change; see lp:1260171).

Robie Basak (racb)
Changed in juju-core (Ubuntu):
status: New → Triaged
importance: Undecided → Medium
Andrew Wilkins (axwalk)
Changed in juju-core:
assignee: nobody → Andrew Wilkins (axwalk)
status: Triaged → In Progress
Revision history for this message
Andrew Wilkins (axwalk) wrote :

Sadly I do not have a 256+ node MAAS to test with. I'm going to mark this as Fix Committed; please let me know if you still have issues.

Changed in juju-core:
status: In Progress → Fix Committed
Curtis Hovey (sinzui)
Changed in juju-core:
status: Fix Committed → Fix Released
Revision history for this message
Felipe Reyes (freyes) wrote :

I'm seeing this problem in juju-core 1.20.8.1 when calling juju destroy-machine, this is a environment where maas has ~580 nodes.

Juju is doing a GET request with a URL 19382 characters, the specific operation is 'list' and it's passing a set of node ids to filter the list (specifically 429 ids)

The problem seems to be in this line https://github.com/juju/juju/blob/master/provider/maas/environ.go#L1158 , it should be doing a POST request.

Sample URL:

GET /MAAS/api/1.0/nodes/?agent_name=f1340a43-06be-4046-8cd8-3b7820e8b0b3&id=node-81858d5e-743f-11e4-8ed9-fcaa141207d4&id=node-f3064ec6-5f33-11e4-ac30-fcaa141207d4&id=node-d7c4c19a-97c0-11e4-823c-fcaa1412....&op=list

The workaround used was to increase the LimitRequestLine[0] in /etc/maas/maas-http.conf

[0] http://httpd.apache.org/docs/2.2/mod/core.html#limitrequestline

tags: added: cts
Jorge Niedbalski (niedbalski)
Changed in juju-core (Ubuntu):
status: Triaged → Confirmed
Curtis Hovey (sinzui)
Changed in juju-core (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Ante Karamatić (ivoks) wrote :

@Curtis, we are seeing this with juju 1.22.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.