juju deploy bundle does not honor existing machine definitions

[Adam Stokes]

TL;DR juju deploy bundle will create additional machines for deployment rather than use the predefined machines. In our case we manually create the machines before a deployment in order to provide some customizations to those machines. Our main case is LXD and creating additional profile configs to allow for openvswitch, kvm, etc.

We expected to be able to add N machines, then customize them slightly before deploying a bundle with placements onto them - specifically we expected that a machine named “1” in a bundle would not be created anew if there was already a machine named “1” in the controller we’re deploying to.

This was the behavior from juju-deployer, see: http://bazaar.launchpad.net/~juju-deployers/juju-deployer/trunk/view/head:/deployer/action/importer.py#L100

juju 2.0-beta3 does this instead:

ubuntu@darthbawlz:~⟫ juju status
[Services]
NAME STATUS EXPOSED CHARM

[Units]
ID WORKLOAD-STATUS JUJU-STATUS VERSION MACHINE PORTS PUBLIC-ADDRESS MESSAGE

[Machines]
ID STATE DNS INS-ID SERIES AZ

ubuntu@darthbawlz:~⟫ juju add-machine
created machine 0
ubuntu@darthbawlz:~⟫ juju add-machine
created machine 1
ubuntu@darthbawlz:~⟫ juju status
[Services]
NAME STATUS EXPOSED CHARM

[Units]
ID WORKLOAD-STATUS JUJU-STATUS VERSION MACHINE PORTS PUBLIC-ADDRESS MESSAGE

[Machines]
ID STATE DNS INS-ID SERIES AZ
0 started 10.0.3.68 juju-8d550d1f-62b4-4f6e-8f84-4f94725f2bb4-machine-0 trusty
1 started 10.0.3.151 juju-8d550d1f-62b4-4f6e-8f84-4f94725f2bb4-machine-1 trusty
ubuntu@darthbawlz:~⟫ cat machine-test.bundle.yaml
machines:
  "1":
    series: trusty
services:
  mysql:
    to:
      - "1"
    charm: cs:trusty/mysql-29
    num_units: 1
series: trusty

ubuntu@darthbawlz:~⟫ juju deploy machine-test.bundle.yaml
added charm cs:trusty/mysql-29
service mysql deployed (charm: cs:trusty/mysql-29)
created new machine 2 for holding mysql unit
# ^^^ this here is not what I wanted.
added mysql/0 unit to machine 2
deployment of bundle "machine-test.bundle.yaml" completed

ubuntu@darthbawlz:~⟫ juju status
[Services]
NAME STATUS EXPOSED CHARM
mysql unknown false cs:trusty/mysql-29

[Relations]
SERVICE1 SERVICE2 RELATION TYPE
mysql mysql cluster peer

[Units]
ID WORKLOAD-STATUS JUJU-STATUS VERSION MACHINE PORTS PUBLIC-ADDRESS MESSAGE
mysql/0 unknown allocating 2 Waiting for agent initialization to finish

[Machines]
ID STATE DNS INS-ID SERIES AZ
0 started 10.0.3.68 juju-8d550d1f-62b4-4f6e-8f84-4f94725f2bb4-machine-0 trusty
1 started 10.0.3.151 juju-8d550d1f-62b4-4f6e-8f84-4f94725f2bb4-machine-1 trusty
2 pending pending trusty

[Richard Harding (rharding)]

Bundles are not meant to do this. The machine specification in bundles is all self contained. This is an effort to make bundles useful across users without needing exact pre-defined setup steps before a bundle is useful.

It is not the goal of bundles in core to implement this in the current form.

There is work in the pipeline that will enable us to specify a profile to lxd as a constraint. That might be the path forward on this so that the bundle can contain the constraint and the machines do not have to be pre-setup. The downside is that then the profiles need to be pre-setup so it's still not helping the reshare point, however we might be able to make the bundle be able to ignore the constraint or warn if it can't be met.

[Ryan Beisner (1chb1n)]

I am resurrecting this, despite its wont-fix status on the basis that the legacy juju-deployer tool does allow this, it is a necessary and valid use case, and the native juju deploy has a gap surrounding this.

[Ryan Beisner (1chb1n)]

See https://bugs.launchpad.net/juju/+bug/1662204.

[Ryan Beisner (1chb1n)]

I disagree with the logic which originally wont-fixed this bug.

Disallowing a procedure on the basis that it might not be directly applicable to all piles of metal is invalid in my opinion.

Disallowing the use of a bundle on the basis that it might actually be specific to a particular pile of metal, is also invalid in my opinion.

I can see where one might apply that filter logic to curated bundles in the charm store, but Juju as a tool should not prevent it.

Bundles, in all of Juju history prior to Juju native deploy, have done this quite well.

With juju-deployer, we can take a pre-existing set of machines, lay a bundle on top, with applications mapped to specific machines, and it just works.

Later, we can update that bundle and re-apply (redeploy) it on top of the existing model, and it just works, applying any changed configuration, and deploying any new applications or units, whether placed on new or existing machines, whether in containers or not.

Until juju native deploy accommodates this and similarly well-established use cases, the juju-deployer tool will likely be a thing.

### One use case:
Deploy a complex set of applications (OpenStack), using a bundle, against numerous manual-provider machines (s390x LPARs).

[Anastasia (anastasia-macmood)]

As this has been the behavior of native Juju deploy that was agreed upon, the best that we can do at this stage is to consider it a Feature Request. I am marking it as a Wishlist item.

[Ryan Beisner (1chb1n)]

Noted. Please understand that native deployer will only gain traction vs. juju-deployer if and as these requests also gain traction.

[Richard Harding (rharding)]

To be clear, there were sessions at the latest sprint around how Juju can
provide native support for things that are needed to close the gap
specifically with Mojo, but this item was in that list. The Juju team took
notes out of that and I expect that there will be bits down the road that
will bring this and other bits of functionality into native juju deploy.

On Tue, Feb 7, 2017 at 8:24 AM Ryan Beisner <email address hidden>
wrote:

> Noted. Please understand that native deployer will only gain traction
> vs. juju-deployer if and as these requests also gain traction.
>
> --
> You received this bug notification because you are subscribed to juju-
> core.
> https://bugs.launchpad.net/bugs/1567169
>
> Title:
> juju deploy bundle does not honor existing machine definitions
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charm-test-infra/+bug/1567169/+subscriptions
>

[Ian Booth (wallyworld)]

The rough plan here is to allow a bundle to define placement in terms of logical machine names. This is the same concept as logical network space names (ie want all of these endpoints bound to the same logical group of subnets etc). But what's in the bundle for machine and space names are logical names, whatever makes sense to think about for the bundle. eg there could be a machine inside the bundle called "dbserver". And you cloud use a placement directive to deploy a charm --to dbserver or even --to lxd:dbserver.

At deploy time, you'd use a separate mapping to map the logical names in the bundle to physical machines. So you'd get to do the special setup on Juju machine 3 because that's going to be the dbserver, and then you'd deploy the bundle (warning, not official syntax, just for illustration):

$ juju deploy mybundle.yaml --machines dbserver=3,appserver=4 --spaces public=default

This approach keeps the bundles nice and reusable and provides the flexibility to deploy according to specific requirements. It's a similar approach to what we use for charm storage - eg the charm defines logically that it wants a file system for "logs", and at deploy time you use the --storage directive to map that to a physical storage instance on the machine or cloud.

[Radoslaw (radraw)]

Is there any known workaround for this bug, how to make Canonical kubernetes distribution be installed on predefined machines ? We have machines on VMWare (no administrative rights, except restarts or console access) with Ubuntu 16.04.

[Anastasia (anastasia-macmood)]

Adding in juju-ci-tools as this functionality - deploying bundles to existing machines - needs a functional test.

[Matt Rae (mattrae)]

It would be great to use existing machines/applications in bundles. Complex deployments may take multiple steps and should be able to make use of already deployed machines/applications. Right now the entire deployment must be in a single bundle.

[Simon Kollberg (simonklb)]

I agree this should be supported. Bundles aren't of much use with the manual provider otherwise.

[Matt Rae (mattrae)]

Noting a workaround that may help some people.

Rather than deploying a service to an existing machine, it is possible to deploy to a service to an existing service.

For example if you add the following newservice in your bundle which will deploy to the same machine as existingservice/0.

  newservice:
    charm: "cs:xenial/ubuntu"
    num_units: 1
    to:
    - lxd:existingservice/0

So in order to address particular machines, you can deploy a service to each machine using the ubuntu charm. Then you can address the machine using that service.

[Dmitrii Shcherbakov (dmitriis)]

Matt,

I know it is possible to do service collocation in a single bundle.

Not sure it is the same for new bundles used for the same model:

juju --version
2.2-rc1-xenial-amd64

cat glance-ss.yaml
series: xenial
services:
  glance-simplestreams-sync:
    charm: cs:xenial/glance-simplestreams-sync
    num_units: 1
    to:
    - glance/0
  keystone:
    charm: cs:xenial/keystone
    num_units: 0 # a hack to be able to add a relation to keystone
relations:
  - [ glance-simplestreams-sync, keystone ]

# no application in the bundle
juju deploy glance-ss.yaml
ERROR the provided bundle has the following errors:
placement "glance/0" refers to an application not defined in this bundle

After adding glance to avoid the first issue, we cannot set num_units to something other than 0 as Juju would then allocate a new machine:

cat glance-ss.yaml
series: xenial
services:
  glance-simplestreams-sync:
    charm: cs:xenial/glance-simplestreams-sync
    num_units: 1
    to:
    - glance/0
  keystone:
    charm: cs:xenial/keystone
    num_units: 0
  glance:
    charm: cs:xenial/glance
    num_units: 0
relations:
  - [ glance-simplestreams-sync, keystone ]

However, in this case Juju figures out that there are no glance units defined - just the application:

juju deploy glance-ss.yaml
ERROR the provided bundle has the following errors:
placement "glance/0" specifies a unit greater than the 0 unit(s) started by the target application

[Radoslaw (radraw)]

Here is patch for juju 2.2 implementing --machines option from comment #8

How to apply:

cd $GOPATH/src/github.com/juju/juju
patch -p1 < ~/juju-machines-2.2.patch

Example usage:

juju deploy somebundle.yaml --machines "0=0 1=1 2=2"

This patch has limitation, only works when bundle is specified as a file. I managed to install Canonical Kubernetes Distribution bundle on pre-created machines using Manual cloud. I hope it will be useful for other cases as well.

[Radoslaw (radraw)]

Here is similiar patch as above, but for juju 2.1.

[Dmitrii Shcherbakov (dmitriis)]

Functionality that would be good to have:

* support v4 bundles (with machine definitions) for this feature;
* support collocating services (e.g. place --to <service-name>/<service-id>) in v4 bundles
* reuse machine and service ids as they appear in a newly supplied bundle. The workflow should be:
 - export an existing bundle with current machine and service IDs;
 - merge it with a new bundle via a third-party tool or edit it manually;
 - juju deploy a merged bundle.

* juju should be able to calculate a diff of a current state and new state: non-existing machines with not-yet-allocated numbers should be added (e.g. don't add machine 3 if it is not there but machine 5 is)
* handle new network and storage bindings gracefully: error out if new ones are added for existing applications (e.g. can't dynamically add or update network spaces for an existing application)

---

Use-cases:

* manual provider (machines are added to the juju state directly and deployment is performed afterwards);
* step-by-step deployment for debugging & CI purposes (e.g. deploy compute, storage, networking first and then telemetry; deploy different combinations of services);
* multi-service bundle modifications (deploy newly released services, placement directives and config options are taken from a 'golden' bundle but we are operating on an old one). Currently it's very inconvenient to add new services because config has to be supplied via --config=<yaml-file> and -n and --to options have to be manually generated.

---

Also, it would be good to update this https://github.com/juju/charmstore/blob/v5-unstable/docs/bundles.md#version-4 as the v4 format documented there is a bit different from what is currently implemented.

[Ante Karamatić (ivoks)]

@Tim I believe this is in line with what we talked about last week?

[Ryan Beisner (1chb1n)]

As this fix/feature hits juju devel/edge, we need to exercise it to confirm that it allows us to stop using juju-deployer in the s390x multi-lpar scenario. Ideally, before it hits juju proposed or stable so that juju core can get confirmation and/or feedback.