launch multiple bootstrap nodes

[William Reade]

As progress towards lp:803042, we should have an environment parameter that controls how many [er: "master", or "bootstrap", or whatever we call them] nodes are launched when we bootstrap.

To avoid the work for this bug ballooning out of control, I propose that this should only be required to actually work properly in the single-node case; there are several things we need to handle before we have a fully-working solution, including:

* making sure all the zookeepers know about all the other zookeepers
* making sure provisioning agents work well together
* figuring out what to do when we lose a master node
* probably more

These bullet points are heavily reminiscent of what juju itself is meant to do, but thinking too much about how we'd make juju deploy itself is giving me a headache. Definitely worthy of further thought/discussion...

[Kapil Thangavelu (hazmat)]

Excerpts from William Reade's message of Wed Oct 19 11:43:07 UTC 2011:
> Public bug reported:
>
> As progress towards lp:803042, we should have an environment parameter
> that controls how many [er: "master", or "bootstrap", or whatever we
> call them] nodes are launched when we bootstrap.
>
> To avoid the work for this bug ballooning out of control, I propose that
> this should only be required to actually work properly in the single-
> node case; there are several things we need to handle before we have a
> fully-working solution, including:
>
> * making sure all the zookeepers know about all the other zookeepers
> * making sure provisioning agents work well together
> * figuring out what to do when we lose a master node
> * probably more
>
> These bullet points are heavily reminiscent of what juju itself is meant
> to do, but thinking too much about how we'd make juju deploy itself is
> giving me a headache. Definitely worthy of further thought/discussion...

Hi william this would be pretty nice, but we should probably talk about this. We
need to refactor bootstrap to launch zk and the provisioning agent as a juju
service, so the management of the zk is the same as any other service.

[William Reade (fwereade)]

OK, here's some further thought. I'm suddenly uncertain whether any of this is a good idea or not; it's just my first approaching-serious attempt at thinking about the intersection of juju, zookeeper, and high availability. Comments appreciated.

First of all: launching a bunch of individual machines from the initial client is problematic, because we don't necessarily know instance-id, dns-name, etc. until we've actually launched the instances, but we need to know all the participating machines in order to run a ZK ensemble (and to communicate the instances to all the potential clients).

We could, I suppose, just launch all the machines and subsequently fiddle around with the ZK config on each of them, restarting ZK as necessary, but it strikes me as rather an ugly way to do things... especially since we have a whole pile of code for managing just that sort of interaction.

Now, just after I joined, Gustavo talked about making juju self-hosting, and I think it's time to consider that possibility seriously. Consider the following a tentative proposal, that I fully expect to get shredded by those wiser than me... but that may eventually evolve into a reasonable plan ;).

(preparation)

* [tweak the initialize command so it checks for /initialized and stops immediately if it finds it]
* [If a PA can't currently tell which MA it's colocated with, add some code/state that allows us to find out]

(bootstrap)

* Launch an instance in "master" mode, exactly as we currently do (preconfigured ZK; run initialize, MA, PA as usual).
* When the PA starts, it should automatically deploy a zookeeper half-charm (which I'll call a ZHC) onto the local MA.
** Note: the ZHC would not do anything to actually install zookeeper -- it'd assume it was already running -- but it would know how to deal with its relations changing (by rewriting the config file and bouncing ZK, perhaps? I understand there's stuff to handle this planned for 3.5.0 but that doesn't help us now).

As I understand it, this would give us exactly what we already have, and not break anything (nobody's going to be messing with zookeeper yet). It's the next bit that involves complexity and possible breakage.

(hardening)

* The PA, when it started, also grabbed a lock, which allows it to consider itself the provisioning agent provisioning agent (note PAPA acronym; you'd think it was planned this way from the start). Holding this lock obligates it to keep track of the number of "master" instances, and launch new ones in "master" mode until the right number are available.

(hm, that sounds suspiciously simple, what will actually happen?)

* PA launches a new instance in "master" mode, which imagines the one and only zookeeper to be the one running on the PA that launched it.
* The new instance comes up:
** initialize connects to ZK, finds everything's already sorted, exits happy;
** MA starts up, connects, isn't asked to do anything, is happy;
** PA starts up, connects, and deploys ZHK onto the local MA;
** huge rocks fall from the sky and kill everyone.

(er, wait)

* OK. What the PA needs to do is essentially either deploy or add-unit, depending on whether or not units already exist.
* Each unit ru...

[William Reade (fwereade)]

Addendum: the more I think about it, the more I think the "ZHC" and the PA should be real, proper, no-fooling charms, and that any slight weirdness in bootstrapping is a small price to pay.

[Kapil Thangavelu (hazmat)]

Excerpts from William Reade's message of Wed Oct 19 21:30:32 UTC 2011:
> Addendum: the more I think about it, the more I think the "ZHC" and the
> PA should be real, proper, no-fooling charms, and that any slight
> weirdness in bootstrapping is a small price to pay.
>

+1 on the addendum, still digesting the main course ;-)

[Kapil Thangavelu (hazmat)]

Excerpts from William Reade's message of Wed Oct 19 17:18:17 UTC 2011:
> OK, here's some further thought. I'm suddenly uncertain whether any of
> this is a good idea or not; it's just my first approaching-serious
> attempt at thinking about the intersection of juju, zookeeper, and high
> availability. Comments appreciated.
>
> First of all: launching a bunch of individual machines from the initial
> client is problematic, because we don't necessarily know instance-id,
> dns-name, etc. until we've actually launched the instances, but we need
> to know all the participating machines in order to run a ZK ensemble
> (and to communicate the instances to all the potential clients).
>
> We could, I suppose, just launch all the machines and subsequently
> fiddle around with the ZK config on each of them, restarting ZK as
> necessary, but it strikes me as rather an ugly way to do things...
> especially since we have a whole pile of code for managing just that
> sort of interaction.

Re multi-launch, its possible to distinguish the different machines in a
multi-machine bootstrap (on ec2) via metadata namespaceing, its probably not
portable. while for looking at allocating additional capacity from bootstrap,
the best way i found was to just modify initialize to add additional machine
states.

I'm thinking this might be better off initially as a specification pushed for
review, just so the discussion is captured in a documentation artifact.

>
> Now, just after I joined, Gustavo talked about making juju self-hosting,
> and I think it's time to consider that possibility seriously. Consider
> the following a tentative proposal, that I fully expect to get shredded
> by those wiser than me... but that may eventually evolve into a
> reasonable plan ;).
>
> (preparation)
>
> * [tweak the initialize command so it checks for /initialized and stops immediately if it finds it]

That's fine, but ideally we're not even executing it multiple times. The
perspective isn't master nodes vs. normal nodes. its just a bootstrap node, vs
additional nodes. The services on the bootstrap node are just normal services
from both a management and code perspective.

> * [If a PA can't currently tell which MA it's colocated with, add some code/state that allows us to find out]
>

the pa would get subsumed into a service consisting of itself and the machine
agent. the pa needs additional work to deal with concurrency, preferrably in the
form of fine grain locks taken out as needed on the specific machines being
acted on, else the work is parallel between machines.

> (bootstrap)
>
> * Launch an instance in "master" mode, exactly as we currently do (preconfigured ZK; run initialize, MA, PA as usual).
> * When the PA starts, it should automatically deploy a zookeeper half-charm (which I'll call a ZHC) onto the local MA.
> ** Note: the ZHC would not do anything to actually install zookeeper -- it'd assume it was already running -- but it would know how to deal with its relations changing (by rewriting the config file and bouncing ZK, perhaps? I understand there's stuff to handle this planned for 3.5.0 but that doesn't help us now).
>
> As I understand it, this would...