maas spaces are mis-translated

[Chris MacNaughton]

Juju seems to have a correct understanding of my spaces:

$ juju spaces --format=yaml
spaces:
  internal:
    10.0.50.0/24:
      type: ipv4
      provider-id: "8"
      status: in-use
      zones:
      - ""
  storage:
    10.0.51.0/24:
      type: ipv4
      provider-id: "11"
      status: in-use
      zones:
      - ""
  undefined:
    10.0.4.0/22:
      type: ipv4
      provider-id: "5"
      status: in-use
      zones:
      - ""

but when I try to actually launch an instance onto one of those spaces where units on the base machine aren't bound to spaces, but then add containers that are bound to spaces, Juju requests machines from MAAS with incorrect space data:

machine-0: 19:25:15 TRACE maas request f9: POST http://10.0.4.2:5240/MAAS/api/2.0/machines/?op=allocate, params=agent_name=63e56b3b-db02-4ad2-8115-d76916426f8e&interfaces=magpie%3Aspace%3D-1&tags=nvme&zone=default

Rather than using space=5 for undefined, Juju seems to be sending space=-1 which MAAS returns a 404 on

[Chris MacNaughton (chris.macnaughton)]

Juju is version 2.4.1-bionic-amd64 (installed from snap)
MAAS is version 2.4.0 (6981-g011e51b7a-0ubuntu1~18.04.1) (apt)

[Andres Rodriguez (andreserl)]

So from the looks of it, this is the issue:

1. In the MAAS db, the /lack/ of space has an 'id' of -1.
2. When juju requests a machine and passes interface=blah:space=-1 , with the intention of getting machines in the "lack of space", then MAAS fails.
3. If we were to send interface=blah:space=undefined, it succeeds.

So, to me this is inconsistent behavior because MAAS allows specifying spaces with their ID's in the db, the same way it allows specifying the space with the name. In this case, MAAS is preventing using the '-1', which is the ID for the 'lack of space'.

# Using IDs/name for space undefined (-1)
ubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=-1 dry_run=True
Not Found
ubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=undefined dry_run=True | pastebinit -f diff
http://paste.ubuntu.com/p/yDTXFKVB8V/

# Using IDs/name for a space other than undefined (-1)
ubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=2 dry_run=True | pastebinit
http://paste.ubuntu.com/p/QpDMBxNqkz/
ubuntu@maas00:~$ maas admin machines allocate interfaces=interface:space=testing dry_run=True | pastebinit
http://paste.ubuntu.com/p/hWD5KY2dYP/

[Mike Pontillo (mpontillo)]

I do think it would be better if Juju doesn't send the space=-1 constraint. (It seems fairly meaningless to say "I want to allocate a machine that is not in a space".) It would be preferable to not send the space constraint to MAAS at all, in that case.

An alternative workaround would be to define spaces for each VLAN you want to use with Juju. That way, there is no need to use the 'undefined' space.

It might be nice to support the "give me a machine not in a space" concept in MAAS, but it still doesn't seem like a very meaningful request. The constraints are powerful enough that you should be able to alternatively specify either no constraints at all, or "give me a machine in subnet X.X.X.X/Y", which would be more likely to give you back what you wanted.

[Mike Pontillo (mpontillo)]

That is to say, part of the justification for creating an "undefined" space is that it's not meaningful. For example, if you add a rack controller to MAAS and it's on a "new" network that has not yet been classified into a space, it will be in the 'undefined' (-1) space.

It's weird for Juju to explicitly request a space that hasn't been acknowledged by the MAAS administrator to be something that is meaningful.

[Mike Pontillo (mpontillo)]

Question: how can the containers on a machine be bound to spaces, and yet the machine itself is not on those spaces?

Shouldn't Juju be asking for a machine that is in the spaces required by the eventual containers?

Also, I think a good workaround might be to assign meaningful spaces for the machines Juju is currently requesting with the `-1` space.

I think until we understand what the use case is for requesting the `-1` space, this is Incomplete for MAAS.

[Andres Rodriguez (andreserl)]

How would someone go about requesting machines with no space at all?

On Fri, Aug 3, 2018 at 5:01 PM Mike Pontillo <email address hidden>
wrote:

> That is to say, part of the justification for creating an "undefined"
> space is that it's not meaningful. For example, if you add a rack
> controller to MAAS and it's on a "new" network that has not yet been
> classified into a space, it will be in the 'undefined' (-1) space.
>
> It's weird for Juju to explicitly request a space that hasn't been
> acknowledged by the MAAS administrator to be something that is
> meaningful.
>
> --
> You received this bug notification because you are subscribed to MAAS.
> https://bugs.launchpad.net/bugs/1785314
>
> Title:
> maas spaces are mis-translated
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1785314/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=juju; status=New; importance=Undecided;
> assignee=None;
> Launchpad-Bug: product=maas; milestone=2.5.0; status=New;
> importance=Undecided; assignee=None;
> Launchpad-Bug: product=maas; productseries=2.3; milestone=2.3.5;
> status=New; importance=Undecided; assignee=None;
> Launchpad-Bug: product=maas; productseries=2.4; milestone=2.4.1;
> status=New; importance=Undecided; assignee=None;
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: andreserl chris.macnaughton mpontillo
> Launchpad-Bug-Reporter: Chris MacNaughton (chris.macnaughton)
> Launchpad-Bug-Modifier: Mike Pontillo (mpontillo)
> Launchpad-Message-Rationale: Subscriber (MAAS)
> Launchpad-Message-For: andreserl
>
--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer

[Andres Rodriguez (andreserl)]

I don’t necessarily think this is about juju. I truly feel this is an
issue that needs resolution in MAAS.

How does one go about allocating machines that are not in any of the
“defined” spaces without negating constraints.

If use doesn’t specify a space constraint, it can get any machine with a
space or without a space, and I don’t think it makes sense negating
constraints.

As such, user should be able To allocate a machine by saying “I want a
machine with an undefined space”. And I think the most backward compatible
way of doing that is using -1/undefined.

On Fri, Aug 3, 2018 at 8:15 PM Mike Pontillo <email address hidden>
wrote:

> Question: how can the containers on a machine be bound to spaces, and
> yet the machine itself is not on those spaces?
>
> Shouldn't Juju be asking for a machine that is in the spaces required by
> the eventual containers?
>
> Also, I think a good workaround might be to assign meaningful spaces for
> the machines Juju is currently requesting with the `-1` space.
>
> I think until we understand what the use case is for requesting the `-1`
> space, this is Incomplete for MAAS.
>
>
> ** Changed in: maas
> Status: New => Incomplete
>
> ** Changed in: maas/2.3
> Status: New => Incomplete
>
> ** Changed in: maas/2.4
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to MAAS.
> https://bugs.launchpad.net/bugs/1785314
>
> Title:
> maas spaces are mis-translated
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1785314/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=juju; status=New; importance=Undecided;
> assignee=None;
> Launchpad-Bug: product=maas; milestone=2.5.0; status=Incomplete;
> importance=Undecided; assignee=None;
> Launchpad-Bug: product=maas; productseries=2.3; milestone=2.3.5;
> status=Incomplete; importance=Undecided; assignee=None;
> Launchpad-Bug: product=maas; productseries=2.4; milestone=2.4.1;
> status=Incomplete; importance=Undecided; assignee=None;
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: andreserl chris.macnaughton mpontillo
> Launchpad-Bug-Reporter: Chris MacNaughton (chris.macnaughton)
> Launchpad-Bug-Modifier: Mike Pontillo (mpontillo)
> Launchpad-Message-Rationale: Subscriber (MAAS)
> Launchpad-Message-For: andreserl
>
--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer

[Andres Rodriguez (andreserl)]

And to be clear, I do agree with mike here that for juju it does make sense
to always have a space defined.

On Fri, Aug 3, 2018 at 9:07 PM Andres Rodriguez <email address hidden>
wrote:

> I don’t necessarily think this is about juju. I truly feel this is an
> issue that needs resolution in MAAS.
>
> How does one go about allocating machines that are not in any of the
> “defined” spaces without negating constraints.
>
> If use doesn’t specify a space constraint, it can get any machine with a
> space or without a space, and I don’t think it makes sense negating
> constraints.
>
> As such, user should be able To allocate a machine by saying “I want a
> machine with an undefined space”. And I think the most backward compatible
> way of doing that is using -1/undefined.
>
>
> On Fri, Aug 3, 2018 at 8:15 PM Mike Pontillo <email address hidden>
> wrote:
>
>> Question: how can the containers on a machine be bound to spaces, and
>> yet the machine itself is not on those spaces?
>>
>> Shouldn't Juju be asking for a machine that is in the spaces required by
>> the eventual containers?
>>
>> Also, I think a good workaround might be to assign meaningful spaces for
>> the machines Juju is currently requesting with the `-1` space.
>>
>> I think until we understand what the use case is for requesting the `-1`
>> space, this is Incomplete for MAAS.
>>
>>
>> ** Changed in: maas
>> Status: New => Incomplete
>>
>> ** Changed in: maas/2.3
>> Status: New => Incomplete
>>
>> ** Changed in: maas/2.4
>> Status: New => Incomplete
>>
>> --
>> You received this bug notification because you are subscribed to MAAS.
>> https://bugs.launchpad.net/bugs/1785314
>>
>> Title:
>> maas spaces are mis-translated
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/juju/+bug/1785314/+subscriptions
>>
>> Launchpad-Notification-Type: bug
>> Launchpad-Bug: product=juju; status=New; importance=Undecided;
>> assignee=None;
>> Launchpad-Bug: product=maas; milestone=2.5.0; status=Incomplete;
>> importance=Undecided; assignee=None;
>> Launchpad-Bug: product=maas; productseries=2.3; milestone=2.3.5;
>> status=Incomplete; importance=Undecided; assignee=None;
>> Launchpad-Bug: product=maas; productseries=2.4; milestone=2.4.1;
>> status=Incomplete; importance=Undecided; assignee=None;
>> Launchpad-Bug-Information-Type: Public
>> Launchpad-Bug-Private: no
>> Launchpad-Bug-Security-Vulnerability: no
>> Launchpad-Bug-Commenters: andreserl chris.macnaughton mpontillo
>> Launchpad-Bug-Reporter: Chris MacNaughton (chris.macnaughton)
>> Launchpad-Bug-Modifier: Mike Pontillo (mpontillo)
>> Launchpad-Message-Rationale: Subscriber (MAAS)
>> Launchpad-Message-For: andreserl
>>
> --
> Andres Rodriguez (RoAkSoAx)
> Ubuntu Server Developer
> MSc. Telecom & Networking
> Systems Engineer
>
--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer

[Mike Pontillo (mpontillo)]

There are two ways to request a machine that isn't in a space today:

 * Use the not_space constraint to tell MAAS you don't want a particular space. (For example, you might use not_space=dmz if you'd like to request any machine except one that is in a particular space.)

 * Omit the space constraint altogether. (This is pretty much the same as saying you don't care which space a machine is in, except that you could get a machine that IS in an unwanted space. But the same is true if you say "give me a machine that has an undefined space" - it could also include interfaces that are in an unwanted space. so it's better in this case for users to be explicit about which spaces they DON'T want.)

[Chris MacNaughton (chris.macnaughton)]

My use case was slightly interesting (I guess); I wanted to assign several machines (not all) as machines in specific spaces (internal network). In addition, all machines are attached to a publicly available network (undefined space). When I didn't specify bindings, I ran into deployment issues with Juju unable to decide on a space for the undefined machines. Adding in specific bindings (`"": undefined') resolved those issues but caused Juju to request these machines incorrectly.

I have since actually named the space which resolves the issue but I basically wanted to leave the default network undefined as everybody is already connected to it.

[Mike Pontillo (mpontillo)]

Thanks. I see how this could be a little bit annoying, and I'm sorry for that. But I still lean toward not fixing this issue in MAAS, because it encourages the use of the undefined space. This is not recommended and can cause problems when a MAAS region is expanded into new networks.

[John A Meinel (jameinel)]

Does this mean that the explicit container request was for the "undefined"
space, which is what Juju was passing along to MAAS ?

I agree with "don't treat the undefined space as a real space" from Mike.
If you want to put containers/machines into a space, it needs to be a
defined one (it can be defined as the public open space, but it should have
*some* sort of definition).

On Fri, Aug 10, 2018 at 3:16 AM, Mike Pontillo <email address hidden>
wrote:

> Thanks. I see how this could be a little bit annoying, and I'm sorry for
> that. But I still lean toward not fixing this issue in MAAS, because it
> encourages the use of the undefined space. This is not recommended and
> can cause problems when a MAAS region is expanded into new networks.
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1785314
>
> Title:
> maas spaces are mis-translated
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1785314/+subscriptions
>

[Chris MacNaughton (chris.macnaughton)]

So, I encountered weirdness when I didn't declare any space ("Juju can't choose a space..." blah blah), that resolved by adding:

bindings:
  "": undefined

to be bundle. The downside to this was that, when certain combinations of applications landed on machines, Juju would pass a request to MAAS where space=-1, which MAAS returned a 404 on and left Juju hanging.

[John A Meinel (jameinel)]

So
bindings:
  "": undefined

Certainly seems like something that we really don't want to support. And if you have applications being deployed into containers on machines where there *are* multiple spaces, Juju has no idea where to put the application (and what spaces route from the host machine into the containers).
So we can certainly put a safety in to prevent you from specifying this, but really where do you *want* those applications to be. I don't think you want them to be undefined in the containers

[Chris MacNaughton (chris.macnaughton)]

John,

I agree that "undefined" is an unfortunate name, but it is also the default space in MAAS if the space is un-named. In the case where somebody is trying out MAAS + Juju it is entirely possible that they still have the default, undefined, space as well as, potentially, an additional space. In that case, the spaces in use would be ["undefined", $something_else] .

I agree that actually naming the space would be a better solution but "undefined" is actually a space :)

[John A Meinel (jameinel)]

So reading through this again, the issue is that Juju reads the list of
spaces from MAAS and grabs the ids for those spaces. And the "undefined"
space always has Id = -1. When Juju makes a request of MAAS for a machine,
it always passes the Ids for the spaces rather than the names, and it turns
out that "undefined" can't be passed by Id.
And while certainly the recommendation is to use defined spaces, rather
than undefined, since newly provisioned machines that have not otherwise
been tuned come up as "undefined" it causes a bit of a problem. We want to
push people towards using spaces, but not require them to configure them,
sounds like we're winding ourselves in knots.

On Mon, Jun 17, 2019 at 10:30 AM Chris MacNaughton <
<email address hidden>> wrote:

> John,
>
> I agree that "undefined" is an unfortunate name, but it is also the
> default space in MAAS if the space is un-named. In the case where
> somebody is trying out MAAS + Juju it is entirely possible that they
> still have the default, undefined, space as well as, potentially, an
> additional space. In that case, the spaces in use would be ["undefined",
> $something_else] .
>
> I agree that actually naming the space would be a better solution but
> "undefined" is actually a space :)
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1785314
>
> Title:
> maas spaces are mis-translated
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1785314/+subscriptions
>

[Launchpad Janitor (janitor)]

[Expired for juju because there has been no activity for 60 days.]

[Joseph Phillips (manadart)]

Keeping this one open, as we are working in area at present.

[Björn Tillenius (bjornt)]

I'm assuming the fix for this will be in Juju. If you feel otherwise, please reopen the MAAS task.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.