multiple juju environments in maas

[Szilard Novaki]

It should be possible to create multiple juju environments in a MAAS. Each user has a different MAAS key, moreover the UI explicitly states that "You'll need a separate MAAS key for each Juju environment.". We found that this is not possible, only one environment is allowed on each MAAS.

To reproduce:
 * create two users on MAAS
 * create an environment.yml for the first user, execute juju bootstrap
 * create an environment.yml for the second user, execute juju bootstrap
 * wait for the bootstrap to finish for the first user
 * execute juju status for the first environment. You will get an error message like:

$ juju status
2013-02-13 10:58:01,643 INFO Connecting to environment...
2013-02-13 10:58:01,761 ERROR juju environment not found: machines are not running (/MAAS/api/1.0/nodes/node-a5375926-75bd-11e2-bd72-5254007b377c/)

After tracking down the problem, I found that juju stores a file in MAAS (provider-state, in /var/lib/maas/media/storage dir on MAAS). This stores the id of zookeeper instance (the juju master host). This file is not created per user, so the second bootstrap overwrote it.

I don't know much about MAAS, but I think this is a MAAS problem, as users should not overwrite other user's files.

[Jeroen T. Vermeulen (jtv)]

Thanks for reporting this. Oddly enough we had *just* figured out that there might be such a problem in theory, and now we know!

What we had in mind was to split the namespaces between environments, rather than users. After all the usual case will be that one "user" (may be a MAAS-internal actor rather than a human) uploads a file and another downloads it. But it's clearly the same problem.

[Raphaël Badin (rvb)]

I don't think this is a problem in MAAS. I think this should be fixed in the MAAS provider in Juju: the provider should derive a prefix from a unique ID that is has and use that prefix when it stores/retrieves files in/from MAAS?
Thoughts?

[Gavin Panella (allenap)]

On 14 February 2013 12:45, Raphaël Badin <email address hidden> wrote:
> I don't think this is a problem in MAAS. I think this should be fixed in the MAAS provider in Juju: the provider should derive a prefix from a unique ID that is has and use that prefix when it stores/retrieves files in/from MAAS?
> Thoughts?

It's worrying that a second user was able to overwrite the file of the
first. My first instinct is to think that each user should have an
independent view of the file store that does not intersect with any
other user's view. But, perhaps this behaviour is intentional. What
restrictions are there around the file store? Are super and non-super
users treated differently? If we're not splitting up the namespace by
user, are we doing it in another way (I guess not, hence this bug)?

Fwiw, if we want to stick with a shared namespace then I think
Raphaël's suggestion makes sense.

[Raphaël Badin (rvb)]

Here is an extract from src/maasserver/api.py:

class AnonFilesHandler(AnonymousOperationsHandler):
    """Anonymous file operations.

    This is needed for Juju. The story goes something like this:

    - The Juju provider will upload a file using an "unguessable" name.

    - The name of this file (or its URL) will be shared with all the agents in
      the environment. They cannot modify the file, but they can access it
      without credentials.
    """

[Raphaël Badin (rvb)]

I think we need to clarify what kind of behavior Juju expects from the file store.

[Julian Edwards (julian-edwards)]

On Thursday 14 Feb 2013 13:36:46 you wrote:
> On 14 February 2013 12:45, Raphaël Badin <email address hidden> wrote:
> > I don't think this is a problem in MAAS. I think this should be fixed in
> > the MAAS provider in Juju: the provider should derive a prefix from a
> > unique ID that is has and use that prefix when it stores/retrieves files
> > in/from MAAS? Thoughts?
>
> It's worrying that a second user was able to overwrite the file of the
> first. My first instinct is to think that each user should have an
> independent view of the file store that does not intersect with any
> other user's view. But, perhaps this behaviour is intentional. What
> restrictions are there around the file store? Are super and non-super
> users treated differently? If we're not splitting up the namespace by
> user, are we doing it in another way (I guess not, hence this bug)?
>
> Fwiw, if we want to stick with a shared namespace then I think
> Raphaël's suggestion makes sense.

We don't want a shared namespace - having different users see each other's
files is crazy.

Also bear in mind that we can't change the behaviour of already-release juju
code. This *has* to be fixed in MAAS. Then MAAS will work like all the other
providers Juju deals with.

[Julian Edwards (julian-edwards)]

On Thursday 14 Feb 2013 13:51:43 you wrote:
> I think we need to clarify what kind of behavior Juju expects from the
> file store.

That is an excellent point.

[Raphaël Badin (rvb)]

Here is some context:

The 2 MPs in which we added support for this:
https://code.launchpad.net/~julian-edwards/maas/anon-get-file/+merge/98798
https://code.launchpad.net/~allenap/maas/anon-get-file-follow-up/+merge/98919

The record of what happens to the file storage API when an env is bootstrapped and mysql+mediawiki charms are deployed:
http://paste.ubuntu.com/1657528/

The names of the files stored in the store (same context as above):
http://paste.ubuntu.com/1657539/

[Raphaël Badin (rvb)]

> We don't want a shared namespace - having different users see each other's
> files is crazy.

You're definitely right. That's also how the other providers do it. Unfortunately, we've implemented this the wrong way: a shared namespace is what we have now :/.

We need to change the store so that each user has a independent view of the file store. Also, each file should have a random string attached to it so that it can be accessed (read-only) anonymously via the API with that key (i.e. /api/1.0/files/?get=filename&key=secret-key-specific-to-this-file).

That's rather easy to change but :

> Also bear in mind that we can't change the behaviour of already-release juju
> code. This *has* to be fixed in MAAS. Then MAAS will work like all the other
> providers Juju deals with.

Indeed, that's where the real problem is really. We need to think hard to fix this while still supporting the exiting provider's code.

[Raphaël Badin (rvb)]

I think I have a solution for this. It will imply fixing the existing python provider (but there is no way around that, right now, the files are fetched using "GET /api/1.0/files/?filename=filename&op=get"; if every user has its own store, we need the provider to use an extra piece of information to distinguish the files named "filename" but owned by different users). But the good news is that I think this solution will ensure a smooth transition for all the existing users, i.e. the change we will make to MAAS won't break the existing installations but juju itself (the MAAS provider part) will have to be SRUed for this bug to be fixed.
I'll write down what I have in mind on Monday.

[Julian Edwards (julian-edwards)]

"use an extra piece of information to distinguish the files named "filename" but owned by different users"

This is the kind of thing I had in mind. It will sadly only work for new (or as you say, SRUed) Juju releases though, unless there's some existing piece of info that MAAS can grok about the requests that differentiates users?

[Gavin Panella (allenap)]

lp:~allenap/juju/maas-anon-urls-2 builds on Jeroen's lp:~jtv/juju/maas-anon-urls.

maas-anon-urls-2 is *not* the right approach now, but it is linked here for reference.

[Gavin Panella (allenap)]

Support for the new per-tenant namespace in MAAS needs to be added to the Python version of Juju. lp:~jtv/juju/maas-anon-urls will provide a starting point. The MAAS provider in the Go version of Juju will support per-tenant namespaces upon initial release.