Cannot deploy a dense openstack bundle with native deploy

[Curtis Hovey]

The dense Openstack Liberty bundle from the charmstore cannot be deployed
by Juju 2 on our maas 1.9

Juju 2.0 using native deploy *cannot* bring up Liberty in 120 minutes.
We wait for 60 minutes for agents to come up and another 30 for
workloads. the remain 30 minutes are for log collection and teardown.

Juju 1.25 with deployer can deploy a similar bundle in 50 to 65 minutes.

We see the kvms taking all the CPU for many minutes. There is also
appears to be a queue on the machines to get the apt lock.

We can see that Juju 2.0 using native deploy to bring up smaller
bundles like landscape works. So I believe this crux of the issue is
that native deploy is not tuned for large bundles, juju-deployer is
tuned. Juju-deployer has several internal and external timeouts
and delay to avoid scheduling too much work that will burden the
controller. I don't think Juju 2.0's native support considers this.

We are considering creating our own Liberty bundle with more hosts to learn if
it reduced the spike in CPU and the wait for apt.

[Cheryl Jennings (cherylj)]

Now that deployer is compatible with 2.0, we should test deploying this bundle on 2.0 with deployer.

[Seman (sseman)]

The Deployer is having a similar issue of timing out during deploying a dense Openstack bundle:

log:
http://juju-ci.vapour.ws/job/maas-1_9-OS-deployer-juju-2x/47/consoleFull

[Seman (sseman)]

This issue seems to be isolated to Juju 2. I was be able to deploy the bundle using Juju 1.
http://paste.ubuntu.com/16369496/
http://juju-ci.vapour.ws/job/maas-1_9-OS-deployer-juju-1x/14/

Bundle:
http://pastebin.ubuntu.com/16369522/

[Seman (sseman)]

Attached Juju 1 logs from http://juju-ci.vapour.ws/job/maas-1_9-OS-deployer-juju-1x/14/

[Seman (sseman)]

Attaching a permanent log for Juju 2 run
Console log : http://paste.ubuntu.com/16384397/

[Martin Packman (gz)]

See bug 1555808 for similar problems with openstack bundle.

[Anastasia (anastasia-macmood)]

@Curtis

We think that this bug has been fixed on master as part of other recently landed work. Could we please confirm through QA testing?

[Curtis Hovey (sinzui)]

Juju 2 on maas 1.9 times out. 1.25 does not
    http://juju-ci.vapour.ws/view/Juju%20Revisions/job/maas-1-9-openstack-deployer/938/console
With luck, juju can get all the services deployed, but they are in error.

Juju 2 on maas 2.0 is better. It does bring up the services, but they are in error. We commonly say this and we sort of expect this because juju-deployer has a proven strategy to deploy complex models, juju does not.

Chris Gregan also confirms in his testing that juju 2 cannot deploy an bundle to bring up openstack.

Also, even when all services report they are up, the test then runs a openstack health check. 1.25 passes this. Juju 2 has never gotten to this test.

[Larry Michel (lmic)]

@John and @Curtis, please note that I have recently deployed the OpenStack bundle from the charm store numerous times without any issue using the Juju 2.1 builds. Based on those tests and IIUC, this looks to have already been fixed. With 2.0 release, we haven't specifically deployed the bundle from the charm store; however, we've successfully deployed a similar OpenStack bundle.

[Larry Michel (lmic)]

The previous comment applies to combination >= Juju 2.1 beta3 and Maas >= 2.1.2 releases.

[Anastasia (anastasia-macmood)]

Based on Larry's comments, marking this as Fix committed for 2.1 and 2.0.

As I cannot accessed released milestone, putting it against currently active releases.