[FFe] juju-core 2.0

[Cheryl Jennings]

The juju team is requesting an FFE for juju-core. Juju2 represents API changes and new features and bugfixes for juju. We are NOT requesting a place on any image for xenial. This FFE is intended to cover the acceptance of the updated juju-core package, as well as the new package juju-core-1.25, which is a new version of the existing juju-core package allowing the existing juju1 binary to be installed.

Related FFes
------------

Required for Juju 2.0:

juju-mongodb3.2: bug 1557852
juju-mongodb2.6: bug 1557830
juju-mongo-tools3.2: bug 1558336

(see note below about mongodb for details)

Required consumers:

conjure: bug 1561037
bigdata: bug 1561043
openstack: Already in universe, dep wait on conjure -> juju 2.0
charm-tools: bug 1546776

User-Facing Features / Changes
------------------------------
* New Terminology
* Command Name Changes
* New Juju home directory
* Multi-Model Support Active by Default
* New Bootstrap and Cloud Management Experience
* Native Support for Charm Bundles
* Multi Series Charms
* Improved Local Charm Deployment
* LXC Local Provider No Longer Available
* LXD Provider
* LXD Containers
* Microsoft Azure Resource Manager Provider
* Bootstrap Constraints, Series
* Juju Logging Improvements
* Unit Agent Improvements
* API Login with Macaroons
* MAAS Spaces
* Resources
* Juju Status Improvements
* Relation Get and Set Compatibility
* Support for new AWS M4 Instance Types
* Support for win10 and win2016

The full list of changes can be found here: https://lists.ubuntu.com/archives/juju/2016-March/006922.html
and in the final release notes.

Timeline
--------
We have released 2 alphas, and 3 betas for juju2 so far, in addition 3 alpha builds of 1.26 which became 2.0. All targeted 2.0 features are now implemented, and we anticipate releasing one release candidate before a final stable build. Juju missed uploading any versions of 2.0 to the archive itself, as well as the initial projected date of having beta1 in xenial before feature freeze. However, juju has been released regularly during the development cycle inside the juju ppa. You can see the details on the delivered features and milestones on https://launchpad.net/juju-core/2.0. The final stable build will be ready in time for the xenial release.

Upgrades
--------
Users upgrading from trusty will find their juju version updated. The current juju-core package will be provided as juju-1.25. Update-alternatives will provide support for toggling /usr/bin/juju between juju-1.25 and juju-2.0 binaries.

We have tested to ensure the intended behavior occurs for the following scenarios:

New Xenial:
No juju, apt-get install juju, juju version will be 2.0.
Example using ppa: <http://paste.ubuntu.com/15555629/>

Upgrading Trusty/Wily/old Xenial:
Already installed juju 1.X, upgraded to latest 1.25.4, 2.0 also installed and is the default juju, update-alternatives used to switch between the two.
Example using ppa: <http://paste.ubuntu.com/15555663/>

Risks
-----
Although juju itself is now feature complete for 2.0, the MAAS 2.0 support will require additional work to be fully supported. MAAS 2.0 is currently under development as well, and is an alpha3 at the time of this writing. Juju will need to add support for the final version of MAAS 2.0, and this is a risk of occurring after xenial is released. If so, we expect to release an sru for juju-core soon after xenial releases to provide this support.

Quality / Testing
-----------------
As this version breaks API with 1.0, testing for features regressions as well as fixing old bugs and avoiding new bugs has been important. The juju team and the greater juju community has been testing 2.0 to ensure it’s stable and ready to support all of the 1.0 workloads in addition to making use of the new 2.0 features.

In comparison to juju-1.25:

* Tests improvements including
* MAAS testing improved
* Container networking
* 9 non-voting tests are now voting
* New tests for all 2.0 features
* Powerpc toolchain is vastly improved
* No test regressions!
* S390 builds reliably, is fully tested and is treated the same as other supported architectures
* 114 bugs has been fixed

Juju practices continuous integration and testing of the juju source tree. All voting tests must pass for ubuntu before release. For example, for beta3, you can see all tests are passing on ubuntu. The full details are here: http://reports.vapour.ws/releases/3815.

The community has also been actively involved with testing, providing feedback, and adopting 2.0 throughout the development cycle.

* 250 bugs and wishlist items filed
* 4 bugs fixed by community members
* ~100-125 downloads for each ppa release

Community Adoption
------------------
The community has already begun adopting 2.0, and is dependent upon 2.0 being inside xenial.

* Multiple external customers are depending on 2.0 inside xenial
* Charm store fully supports 2.0 features
* Community charmers
* Gitlab, openstack and other charms already are using 2.0 features
* Many other charms are dependent upon multi-model/controller features
* Public demo and feedback sessions @ charmers summit
* Public talks accepted for discussing juju 2.0 at conferences
* 2 packages (conjure, bigdata) currently in the upload queue have been built and depend upon the new version of juju 2.0.

Packaging
---------
The security team has requested changes to the current juju packaging with the ultimate goal of breaking out the embedded non-juju golang dependencies.

As part of this FFE, the juju-core package depends upon all currently packaged golang depends that are already in the archive.

The remaining ~15 dependencies have been packaged and will be uploaded to the archive once the archive has opened again for Y development. We will not be attempting to add these additional packages as part of this FFE.

Mongodb
-------

Juju only requires a mongodb for running a controller. With the replacement of the local provider with the lxd provider, we no longer need to install mongodb on the user's machine for client deploys.

We do still want a juju-mongodb package in the archive, but only for 64-bit arches that we support running controllers on. We can build armhf (and even i386) juju clients without changing that.

Juju as proposed will run against either the current juju-mongodb in the archive which is 2.4 and no longer supported upstream, or 3.X if it's available. So, while we don't strictly depend on the proposed juju-mongodb packages, they are how we plan to get to a version of mongo we can support.

Future release planning
------------------------

Given lessons learned this cycle the team has determined it's in the interests of everyone to perform the following next cycle:

- continue to break out the Go dependencies. There are 10 additional ones that make sense and that the team has verified can be pulled out. They're prototype packages currently and will be made part of the next cycle deliverable.

- Juju has been training an employee working to gain upload rights to Juju. This will include another team member and we will work so that they have upload rights to the package set around all of Juju and the community including charm and charm-tools.

- The juju core team will upload the current release of Juju upon the opening of the archive for 16.10. There after all beta releases will also be uploaded to the archive as well to the devel PPA. This is now possible due to the team responding to the feedback in this current process attempt.

[Martin Pitt (pitti)]

Launchpad Bug Tracker [2016-02-17 19:37 -0000]:
> - Removal of LXC from 2.0 codebase

Does that mean "in favor of LXD", or "juju-local will be dropped"? I
guess/hope the latter, but making sure :-)

[Stéphane Graber (stgraber)]

That's in favor of LXD.

All LXC code from JuJu has been or is in the process of being rewritten to use LXD instead.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Richard Harding (rharding)]

As part of our exception, we're blocking the inclusion of two other packages:

https://bugs.launchpad.net/ubuntu/+bug/1561043
https://bugs.launchpad.net/ubuntu/+bug/1561037

As we update Juju around this exception those will be effected and will need to have some leeway in the process.

[Adam Stokes (adam-stokes)]

Also want to mention the openstack package as well.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Martin Packman (gz)]

Updated the description with the example upgrade testing using the test packages in the ppa.

[Dimitri John Ledkov (xnox)]

local-kvm provider is not available in the beta, and is a regression compared with juju-core 1.25. Is there going to be a local kvm provider? (or e.g. libvirt provider on par with lxd provider)

[Richard Harding (rharding)]

No, there is no KVM local provider in Juju 2.0. Product management has made the determination that the lxd tool provides the needed functionality for both the lxc and kvm use cases. lxd in Juju 2.0 will be the answer.

[Adam Stokes (adam-stokes)]

xnox, there is this bug: https://bugs.launchpad.net/juju-core/+bug/1547665

[Nicholas Skaggs (nskaggs)]

Adam has built the initial version desired for upload: https://launchpad.net/~adam-stokes/+archive/ubuntu/juju-pkg

[Jamie Strandboge (jdstrand)]

"The security team has requested changes to the current juju packaging with the ultimate goal of breaking out the embedded non-juju golang dependencies.

As part of this FFE, the juju-core package depends upon all currently packaged golang depends that are already in the archive.

The remaining ~15 dependencies have been packaged and will be uploaded to the archive once the archive has opened again for Y development. We will not be attempting to add these additional packages as part of this FFE."

I looked at the Packages file in the PPA and verified that juju was Built-Using the specified packages. One small thing, juju Build-Depends on golang-go.net-dev but this is a transitional package that pulls in golang-x-net-dev (which is found in Built-Using). Please adjust the Build-Depends to use golang-x-net-dev instead.

With my security team hat on, progress was made on bug #1508120 with the current packaging in the ppa so juju is heading in the right direction wrt to embedded code copies. Therefore the security team will not block this FFe. Thank you for making these changes.

With my MIR team hat on I'll comment on the embedded code copes, conditional ACK provided bug #1508120 is updated to enumerate the remaining ~15 dependencies and the plan to address them.

[Martin Packman (gz)]

I've edited the bug the clarify the juju-mongodb situation. There's a change to both tweak the version of mongodb and make the unit tests more reliable, which was held from landing for beta3 waiting on the package in ubuntu. It will be included in the rc.

[Richard Harding (rharding)]

In an IRC conversation with: stgraber, infinity, mgz

We agreed that the outstanding issues are the following:

- adding a comment to the juju 2.0 FFe that clarifies that the juju client is built and runs on all architectures, including 32bit.
- move the juju binary metapackage suggest juju-1.25 so that upgrades will make sure that juju 1 stays available
- make juju1 the higher update-alternatives priority.
- trim the binary size of the juju package by reducing the duplication
- investigate the ability to perform stripping to reduce size

[Richard Harding (rharding)]

Add item

- document the process for avoiding a late stage FFe and package upload for 16.10

[Richard Harding (rharding)]

- adding a comment to the juju 2.0 FFe that clarifies that the juju client is built and runs on all architectures, including 32bit.

Comment added under the heading Mongodb to this effect since that's where the confusion derived from.

- document the process for avoiding a late stage FFe and package upload for 16.10

Added section to the FFe under the heading "Future release planning"

- move the juju binary metapackage suggest juju-1.25 so that upgrades will make sure that juju 1 stays available

This change is made and new builds are being done.

- make juju1 the higher update-alternatives priority.

We cannot make this change due to direction from product management. We have filed this for record under https://bugs.launchpad.net/ubuntu/+source/juju-core/+bug/1564670.

We've also created the Juju bug to perform the update that outputs the mitigating messaging to help the user in the 2.0 Juju.
https://bugs.launchpad.net/juju-core/+bug/1564622

- trim the binary size of the juju package by reducing the duplication

See bug https://bugs.launchpad.net/juju-core/+bug/1564677 for the work to perform this. This will be a goal of the next full release to perform this action to reduce the Juju binary size.

- investigate the ability to perform stripping to reduce size

See bug https://bugs.launchpad.net/juju-core/+bug/1564662 for this work. This will be a goal for the next full release to perform this action and reduce the Juju binary size. However, we need time to validate that it is 100% safe and we feel the risk is too high to push it through today.

[Richard Harding (rharding)]

These updates were presented to slangasek and the point of the making juju1 the default is the one point left that we requested feedback on.

Feedback from slangasek is that if we're not prepared to make juju1 the default then:
"upon upgrade, juju == juju2; if you want juju1, run juju-1.$fwibble instead"

The Juju team defers to the distro team in regards to Juju upgrades and will move forward with this plan during updates and update the juju2 command such that if it detects you're a new juju2 user with existing juju1 environments.yaml file we will direct you to the new command to run as a helpful hint.

Please direct feedback on this decision to slangasek.

[Martin Packman (gz)]

The proposed packaging updates are now available in Adam's PPA:

<https://launchpad.net/~adam-stokes/+archive/ubuntu/juju-pkg>

This includes:

* Switch from golang-go.net-dev to golang-x-net-dev as mentioned by ~jdstrand
* Use of Suggests to keep Juju 1.X tools around through upgrade from ~adconrad

[Steve Langasek (vorlon)]

A quick note that the juju-core-1 package has been reviewed in the NEW queue and rejected for a number of small and easy-to-fix, but critical to NEW processing, issues. Email has been sent to Adam and Martin with details for what needs to be fixed in a resubmit.

[Steve Langasek (vorlon)]

juju-core 2.0 introduces new build dependencies on the following packages in universe:

  golang-github-azure-azure-sdk-for-go
  golang-golang-x-oauth2
  golang-google-api
  golang-gopkg-mgo.v2

Have these already undergone security review? I checked the first of these and found no MIR bug for the package.

[Martin Packman (gz)]

Packaging branches have been updated based on review feedback, new versions need uploading:

lp:~juju-qa/ubuntu/xenial/juju/juju-1.25.4
lp:~juju-qa/ubuntu/xenial/juju/xenial-2.0-beta3

[Richard Harding (rharding)]

Steve, comment #12 seems to imply we're ok with the way with the current set. We've got a bug and task to split those out for Y. Did you want another ok from Jamie on this?

[Steve Langasek (vorlon)]

Rick,

I have double-checked with Jamie and Tyler from the Security Team. You are doing the right thing by build-depending on those modules that are available as separate packages in xenial, thank you for this. However, this does not mean that these packages you build-depend on can go into main without going through the MIR process. To the contrary, adding the build-dependency is the trigger that lets us know that the packages *need* to go through the MIR process (and in particular, the security review of these modules). Otherwise, the juju team could in theory add new bundled modules indefinitely to the source without ever getting Security Team visibility on that code.

This will not block the feature freeze exception for juju-core 2.0, we will continue moving ahead with that in parallel, but we do need the juju team to start that MIR process for these new universe build-deps so that they can be properly reviewed prior to 16.04 release.

[Steve Langasek (vorlon)]

For a more detailed account of the Security Team's position on MIRs for these bundled deps, see: https://bugs.launchpad.net/ubuntu/+source/juju-mongodb/+bug/1267393/comments/68

[Steve Langasek (vorlon)]

Bugs (in addition to the missing MIRs):
  - juju-2.0 is shipping a link named 'juju-upgrade-mondo' instead of 'juju-upgrade-mongo'. This probably needs fixing.
  - juju-2.0 uses and src/github.com/lxc/lxd/ (expected) and does not build-depend on the golang-github-lxc-lxd-dev package in the archive. This definitely needs fixing; this will be tracked in bug #1568148.

Recommendations:
  - debian/tests/control: referencing '@' instead of 'juju-2.0' makes this more future-proof against further binary package name changes.

Questions:
  - Nothing from juju-core 2.0 source depends on juju-mongodb (which will be mongodb 3.2 for 16.04; bug #1557852). When I asked on IRC why this is, I was told that it's because this is only needed on the server, whereas the juju-2.0 package is the client. I pointed out that the juju-2.0 package ships jujud, which is server, not client. I was told that there is an open bug discussing removing jujud from the client package, which is only used with the --upload-tools option (not recommended and only intended for developers). And juju-upgrade-mongo is a client plugin that lets the client request an upgrade remotely of the server.

And recording notes here for the benefit of my future self.

The current binary packages with relationships are (in trusty and xenial):

  - juju Depends: juju-core
  - juju-local Depends: juju-core
  - juju-local-kvm Depends: juju-core
  - juju-core is the base package containing the binaries (/usr/lib/juju-1.25.0/bin) and calling update-alternatives for /usr/bin.

The updated packages are:
  - juju-core Depends: juju-1
  - juju-local Depends: juju-1
  - juju-local-kvm Depends: juju-1
  - juju-1 is the base package for juju-core-1 containing the binaries (/usr/lib/juju-1.25.0/bin) and the /usr/bin/juju-1 wrapper script, and does not call update-alternatives.
  - juju-2.0 is the base package for juju-core containing the binaries (/usr/lib/juju-2*/bin), the /usr/bin/juju-2 wrapper script, and the /usr/bin/juju symlink, and does not call update-alternatives. It also declares a Breaks: juju-core (<= 1.25.4), which is the version of this package which calls update-alternatives. This forces the old juju-core to be deconfigured (and the /usr/bin/juju link removed) before juju-2.0 is unpacked, and ensures that there is no overlap on the filesystem.
  - juju Depends: juju-2.0 and Suggests: juju-core.

On upgrade, if the user has the juju package installed, the result is:
  - juju-core is already installed as a dependency; this dependency is dropped to a suggests: on upgrade, but that's enough to keep the package installed, and upgraded to pull in juju-1.
  - juju-2.0 is pulled in as a new dependency.
  - juju-2.0 and juju-1 commands are provided on the path. unversioned juju-* commands are also provided on the path, pointing to /usr/lib/juju-2.0 implementations.

On upgrade, if the user does not have the juju package installed, only juju-core, the result is:
  - the user gets juju-1 on the path. juju-2.0 is not installed. This is probably acceptable because 'apt install juju' is the recommended interface for installing juju and juju-core is mostly an implementation detail. There is no good ...

[Nicholas Skaggs (nskaggs)]

For juju, we still need to ensure our MIR's are reviewed and land into main, and that our packaging utilizes all possible dependencies that are already in the archive. This work is targeted before release and is being tracked in bug 1568148.

[Martin Packman (gz)]

> - juju-2.0 is shipping a link named 'juju-upgrade-mondo' instead of
> 'juju-upgrade-mongo'. This probably needs fixing.

Fixed in the packaging branch.

> - juju-2.0 uses and src/github.com/lxc/lxd/ (expected) and does not
> build-depend on the golang-github-lxc-lxd-dev package in the archive. This
> definitely needs fixing; this will be tracked in bug #1568148.

This is a deliberate omission for beta3. Since we did the tarball release, there have been two new lxd rcs uploaded that change their api and break compatibility. It will be included with our rc packaging, by which time they will hopefully be stable.

[Steve Langasek (vorlon)]

juju-core 2.0~beta3-0ubuntu3 and juju-core-1 1.25.4-0ubuntu5 have both been accepted into the Ubuntu archive. However, they are stalled in xenial-proposed because the juju-core autopkgtests have regressed (on amd64, i386, and ppc64el):

http://autopkgtest.ubuntu.com/packages/j/juju-core/xenial/amd64/

I know that the packaging delta from juju-core 1.25.0-0ubuntu3 included new "lxd" autopkgtests replacing the "local" autopkgtests, which makes sense. But these tests don't run, something was lost in translation from the old tests:

adt-run [16:49:12]: test current-lxd-provider: [-----------------------
+ sh debian/tests/normal-user.sh debian/tests/lxd-provider
+ adduser --disabled-password --gecos jujutest
adduser: Only root may add a user or group to the system.
adt-run [16:49:13]: test current-lxd-provider: -----------------------]

The old 'local' tests declared a dependency on sudo and declared that they needed to run as root. The new lxd test has dropped this information, and therefore fails as above.

Please see the logs linked at the above url for the full details on failures; the failure seems to be the same for all three failing tests (current-lxd-provider, current-manual-provider, future-lxd-provider) across all three affected archs (amd64, i386, ppc64el). Please test your fix with autopkgtest (e.g., 'adt-run -B --unbuilt-tree . --- adt-virt-$option') before reuploading to the archive.

[Martin Packman (gz)]

Sadly, being stuck in -proposed is probably correct for beta3, due to the above mentioned lxd breakages.

Note that the future-lxd-provider gets futher, but still breaks due to changes in lxd since the test was last modified with beta2:

+ lxd-images import ubuntu yenial --alias ubuntu-yenial
debian/tests/lxd-provider: line 8: lxd-images: command not found

Also, we don't have coverage for lxd on armhf or s390x as they can't provide machine-level isolation. That's probably fine, but can maybe be improved when we have more confidence lxd/juju won't blow things up as much as the dev versions have tended to.

[Steve Langasek (vorlon)]

On Sat, Apr 09, 2016 at 10:32:43PM -0000, Martin Packman wrote:
> Sadly, being stuck in -proposed is probably correct for beta3, due to
> the above mentioned lxd breakages.

It may be the case that the package should be stuck in -proposed, but not
for these test failures, which are very clearly bugs in the tests and not in
the juju-core code. The tests, not the lxd code, call adduser and fail
because they have not declared that root privileges are needed.

It is also not the expectation that known-broken packages be uploaded to
xenial-proposed and allowed to linger there. Please fix these tests ASAP
and reupload.

> Note that the future-lxd-provider gets futher, but still breaks due to
> changes in lxd since the test was last modified with beta2:

> + lxd-images import ubuntu yenial --alias ubuntu-yenial
> debian/tests/lxd-provider: line 8: lxd-images: command not found

Yes, you're right, I overlooked that the error message here was different -
sorry about that. This is also a test bug (because incompatible with
current lxd-client) that needs fixing. If I'm not mistaken, the current
syntax for this is 'lxc image import ubuntu $SERIES --alias ubuntu-$SERIES'.

Also, the tests are missing a declaration for their dependency on the
lxd-client package. It definitely won't be installed in the autopkgtest
testbed without this declaration.

> Also, we don't have coverage for lxd on armhf or s390x as they can't
> provide machine-level isolation. That's probably fine, but can maybe be
> improved when we have more confidence lxd/juju won't blow things up as
> much as the dev versions have tended to.

Unless the tests themselves actually require machine-level isolation, I
would suggest that the 'breaks-testbed' restriction that's been declared is
sufficient, and there's no need to specify 'isolation-machine' as well - and
that you can leave it to the autopkgtest implementors to ensure that the
testbed is appropriately isolated.

(If Martin Pitt has specifically requested that you use the
'isolation-machine' restriction, then I defer to him.)

[Martin Pitt (pitti)]

Steve Langasek [2016-04-11 6:29 -0000]:
> > + lxd-images import ubuntu yenial --alias ubuntu-yenial

Also, no such release :-)

Do you actually need to do this? lxd automatically downloads images
from the remote when needed.

> > Also, we don't have coverage for lxd on armhf or s390x as they can't
> > provide machine-level isolation. That's probably fine, but can maybe be
> > improved when we have more confidence lxd/juju won't blow things up as
> > much as the dev versions have tended to.
>
> Unless the tests themselves actually require machine-level isolation, I
> would suggest that the 'breaks-testbed' restriction that's been declared is
> sufficient, and there's no need to specify 'isolation-machine' as well - and
> that you can leave it to the autopkgtest implementors to ensure that the
> testbed is appropriately isolated.

If the tests work in LXC (i. e. the whole thing works in nested
containers), then restriction-container is enough. But please test
this locally before. Otherwise, if the tests fail with the lxc runner
but work in qemu, keep the i-machine for now.

Not sure what the "breaks-testbed" does, but this does nothing wrt.
selecting/skipping tests between lxc and full VMs. It merely means
that a new instance/container will be created after running this test
(and also that the test will be skipped with the "null" runner, but
that's irrelevant here).

[Martin Packman (gz)]

Sorry, should have been clearer. Yes, I am working on fixing those issues with the tests. We're also trying to get a new source release this week that can be put the the archive, this package can't move forward without that.

"LXD v2.0.0-rc8 does not work with Juju v2.0-beta3"
<https://lists.ubuntu.com/archives/juju-dev/2016-April/005339.html>

Also as of this morning our unit tests on master break because the xenial daily image now includes lxd 2.0.0 rather than rc9, which again changes behaviour.

Martin Pitt: We currently don't have lxd as a dep on this package because it's included by default on the image. If I add it (as a depends, or recommends), does that mean our autopkgtests including the lxd provider ones will run on new lxd package uploads and prevent them entering the archive in future?

The image import can likely be dropped, the 'yenial' is a test about catching breakage due to a new ubuntu development release, but we probably don't need to actually start lxd containers in the fake new release for the coverage we want.

Thanks for the info on the meaning of the various flags. For now I think we do need isolation-machine (keep having bugs where networking interfaces etc get created wrongly) but will look at just using restriction-container later.

[Martin Packman (gz)]

It seems juju-core-1 is just stuck on the juju-mongodb2.6 package lying around unreviewed in the new queue:

<http://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#juju-core-1>

juju-core-1 (- to 1.25.4-0ubuntu5)
  Maintainer: Ubuntu Developers
  Section: universe/devel
  3 days old
  juju-local/amd64 unsatisfiable Depends: juju-mongodb2.6
  juju-local-kvm/amd64 unsatisfiable Depends: juju-mongodb2.6
  Not considered

Is there a reason that hasn't been tackled yet?

[Martin Packman (gz)]

...and I refresh the page and see someone has now reviewed both mongodb packages. Thanks!

[Steve Langasek (vorlon)]

On Tue, Apr 12, 2016 at 01:50:28PM -0000, Martin Packman wrote:
> Martin Pitt: We currently don't have lxd as a dep on this package
> because it's included by default on the image.

This is not a justification for omitting the dependency. If the juju
package relies on lxd for its core functionality, it should be declared as a
dependency, even if the lxd package is installed by default. The only
exceptions to this rule are packages declared Essential: yes.

> If I add it (as a depends, or recommends), does that mean our autopkgtests
> including the lxd provider ones will run on new lxd package uploads and
> prevent them entering the archive in future?

Yes. (Which, I hope it's clear, is the desired behavior for autopkgtests.)

[Martin Packman (gz)]

Have updated packaging and upstream for both juju 2.0 and 1.25 ready to upload.

https://code.launchpad.net/~juju-qa/ubuntu/xenial/juju/juju-1.25.5

New 1.25.5 bugfix release. Fixed the adt tests, including making them run with isolation-container rather than isolation-machine.

adt-run [18:19:14]: @@@@@@@@@@@@@@@@@@@@ summary
client PASS
current-local-provider PASS
future-local-provider PASS
current-manual-provider PASS
future-manual-provider PASS

Final note, I have for now reverted the dependency change to use juju-mongodb2.6 as upstream code doesn't correctly detect its existence and continues to use juju-mongodb (2.4) anyway. I will follow up with a bug against 1.25.6 for this.

https://code.launchpad.net/~juju-qa/ubuntu/xenial/juju/xenial-2.0-beta4

New 2.0-beta4 release. Fixed a bunch of issues with the tests, including the requirement for manual lxd configuration. Unfortunately, lxd currently doesn't work in the nested container setup of these tests, so I have not been able to verify everything works yet.

adt-run [23:53:12]: @@@@@@@@@@@@@@@@@@@@ summary
current-lxd-provider SKIP Test requires machine-level isolation but testbed does not provide that
client PASS
current-manual-provider PASS
future-manual-provider PASS

The juju-2.0 package now recommends lxd as suggested.

[Martin Packman (gz)]

Well, mixed news.

The 1.25.5 upload yesterday got bopped by the keyserver failing, but has now been reuploaded and we're waiting on the autopkgtest run.

The 2.0-beta4 upload happened, and we have test results:

<http://autopkgtest.ubuntu.com/packages/j/juju-core/>

Things are in a better state (s390x is actually green) but the substrates that can run the lxd test all failed with:

<https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/amd64/j/juju-core/20160415_045320@/log.gz>

+ juju bootstrap my-controller lxd --upload-tools --debug --config default-series=xenial
2016-04-15 04:44:46 INFO juju.cmd supercommand.go:60 running juju [2.0-beta4 gc go1.6.1]
2016-04-15 04:44:46 INFO cmd cmd.go:141 cloud "lxd" not found, trying as a provider name
2016-04-15 04:44:46 INFO cmd cmd.go:141 no credentials found, checking environment
2016-04-15 04:44:46 DEBUG juju.cmd.juju.commands bootstrap.go:365 preparing controller with config: map[type:lxd name:admin uuid:2bf2d716-0a36-46a3-8b77-83dc8fc66507 controller-uuid:2bf2d716-0a36-46a3-8b77-83dc8fc66507 default-series:xenial]
2016-04-15 04:44:50 DEBUG juju.tools.lxdclient client.go:73 connecting to LXD remote "local": ""
2016-04-15 04:44:52 DEBUG juju.tools.lxdclient client.go:73 connecting to LXD remote "juju-remote": "10.0.8.1"
2016-04-15 04:44:52 ERROR cmd supercommand.go:448 Get https://10.0.8.1:8443/1.0/profiles: Forbidden
adt-run [04:44:53]: test current-lxd-provider: -----------------------]
adt-run [04:44:53]: test current-lxd-provider: - - - - - - - - - - results - - - - - - - - - -
current-lxd-provider FAIL non-zero exit status 1

I'm not sure what is different here, from the setup I got to pass successfully, using a canonistack machine:

<http://paste.ubuntu.com/15852422/>

The 30 mins fetching stuff from the archive into the container there is... suspect, but also well after the autopkgtest.ubuntu.com setup fails.

I'm not sure how to debug further, given I can't exactly reproduce the real setup.

Additionally, and easier to fix, the armhf future-manual-provider test fails because it can't install juju-mongodb3.2:

<https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/armhf/j/juju-core/20160415_091616@/log.gz>

2016-04-15 09:16:01 DEBUG juju.cmd.jujud bootstrap.go:322 starting mongo
2016-04-15 09:16:01 DEBUG juju.cmd.jujud bootstrap.go:347 calling ensureMongoServer
2016-04-15 09:16:01 INFO juju.mongo mongo.go:394 Ensuring mongo server is running; data directory /var/lib/juju; port 37017
2016-04-15 09:16:01 INFO juju.mongo mongo.go:559 installing [juju-mongodb3.2]
2016-04-15 09:16:01 INFO juju.utils.packaging.manager utils.go:57 Running: apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet install juju-mongodb3.2
2016-04-15 09:16:02 ERROR juju.utils.packaging.manager utils.go:103 packaging command failed: encountered fatal error: unable to locate package; cmd: "apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet install juju-mongodb3.2"; output: Reading p...

[Martin Packman (gz)]

The juju-core-1 autopkgtests are all green:

<http://autopkgtest.ubuntu.com/packages/j/juju-core-1/>

We have bug 1570759 reported for the slightly weird state of the xenial archive with 1.25.5 in main but 2.0-beta4 stuck in proposed.

I filed a bunch of bugs.

The autopkgtest problems for 2.0: bug 1571082, bug 1571072
The mongodb dependency for 1.25: bug 1570650
Usability stuff with manpages and bash completion: bug 1570654, bug 1570651, bug 1570660, bug 1570657

There's also some potential for confusion for people who've installed juju betas from our development ppa this cycle, that some additional Breaks/Replaces may help with.

[Launchpad Janitor (janitor)]

This bug was fixed in the package juju-core - 2.0~beta4-0ubuntu2

---------------
juju-core (2.0~beta4-0ubuntu2) xenial; urgency=medium

  * Update debian/tests/ to work in the production network environment
    (LP: #1571082).

 -- Michael Hudson-Doyle <email address hidden> Tue, 19 Apr 2016 21:20:58 +1200