Please add support for utopic

[James Page]

Local provider currently fails on utopic:

Testing juju bootstrap:
2014-04-30 14:27:11 INFO juju.cmd supercommand.go:297 running juju-1.18.1-utopic-amd64 [gc]
2014-04-30 14:27:11 DEBUG juju.environs.configstore disk.go:64 Making /home/ubuntu/.juju/environments
2014-04-30 14:27:12 INFO juju.provider.local environprovider.go:147 checking state port
2014-04-30 14:27:12 INFO juju.provider.local environprovider.go:147 checking API port
2014-04-30 14:27:12 INFO juju.provider.local environprovider.go:40 opening environment "local-test"
2014-04-30 14:27:13 DEBUG juju.provider.local environ.go:282 found "10.0.3.1" as address for "lxcbr0"
2014-04-30 14:27:13 INFO juju.environs.bootstrap synctools.go:34 checking that upload is possible
2014-04-30 14:27:13 INFO juju.cmd cmd.go:113 uploading tools for series [utopic precise]
2014-04-30 14:27:13 DEBUG juju.environs.sync sync.go:276 Building tools
2014-04-30 14:27:13 DEBUG juju.environs.tools build.go:115 looking for: juju
2014-04-30 14:27:13 DEBUG juju.environs.tools build.go:156 checking: /usr/bin/jujud
2014-04-30 14:27:13 INFO juju.environs.tools build.go:162 found existing jujud
2014-04-30 14:27:13 INFO juju.environs.tools build.go:172 target: /tmp/juju-tools961712907/jujud
2014-04-30 14:27:13 DEBUG juju.environs.tools build.go:223 forcing version to 1.18.1.1
2014-04-30 14:27:13 DEBUG juju.environs.tools build.go:39 adding entry: &tar.Header{Name:"FORCE-VERSION", Mode:436, Uid:0, Gid:0, Size:8, ModTime:time.Time{sec:63534464833, nsec:0x1914e9e8, loc:(*time.Location)(0x18b8840)}, Typeflag:0x30, Linkname:"", Uname:"ubuntu", Gname:"ubuntu", Devmajor:0, Devminor:0, AccessTime:time.Time{sec:63534464833, nsec:0x1914e9e8, loc:(*time.Location)(0x18b8840)}, ChangeTime:time.Time{sec:63534464833, nsec:0x1914e9e8, loc:(*time.Location)(0x18b8840)}, Xattrs:map[string]string(nil)}
2014-04-30 14:27:13 DEBUG juju.environs.tools build.go:39 adding entry: &tar.Header{Name:"jujud", Mode:493, Uid:0, Gid:0, Size:36841744, ModTime:time.Time{sec:63534464833, nsec:0x1914e9e8, loc:(*time.Location)(0x18b8840)}, Typeflag:0x30, Linkname:"", Uname:"ubuntu", Gname:"ubuntu", Devmajor:0, Devminor:0, AccessTime:time.Time{sec:63534464833, nsec:0x1914e9e8, loc:(*time.Location)(0x18b8840)}, ChangeTime:time.Time{sec:63534464833, nsec:0x1914e9e8, loc:(*time.Location)(0x18b8840)}, Xattrs:map[string]string(nil)}
2014-04-30 14:27:24 INFO juju.environs.sync sync.go:295 built tools 1.18.1.1-utopic-amd64 (7193kB)
2014-04-30 14:27:24 DEBUG juju.environs.sync sync.go:200 Uploading tools for [utopic precise]
2014-04-30 14:27:24 DEBUG juju.environs.sync sync.go:230 generating tarballs for [utopic precise]
2014-04-30 14:27:24 INFO juju.cmd cmd.go:113 Bootstrap failed, destroying environment
2014-04-30 14:27:24 ERROR juju.cmd supercommand.go:300 invalid series "utopic"

[Curtis Hovey (sinzui)]

I think this issue can be fixed by
1. make utopic packages for 1.18.1 in the stable ppa
2. run assemble-public-tools for 1.18.1 again
3. publish the tools
4. tell streams.canonical.com to publish 1.18.1

But how do the packages in ports become tools? The assembly script knows utopic is the devel series, and series-ambiguous package will be classified as a utopic tool. so streams.canonical.com might make new tools using the old packages.

[John A Meinel (jameinel)]

I believe this is more about the local provider, which doesn't actually
look for tools on streams.canonical.com

I don't quite understand why we don't recognize 'utopic', though, as we
*should* be reading /usr/share/distro-info/ubuntu.csv to find what series
are available.

Certainly you can see from the debug output that it is creating 'utopic'
tools, it just gets to a point where it ends up deciding that it isn't
actually a valid series.

Maybe the issue is actually in the LXC code? Certainly we found that on
precise we had to install ubuntu-cloud-image packages so that it could know
what "trusty" is.

On Thu, May 1, 2014 at 1:56 PM, Robie Basak <email address hidden>wrote:

> ** Changed in: juju-core (Ubuntu)
> Status: New => Triaged
>
> --
> You received this bug notification because you are subscribed to juju-
> core.
> https://bugs.launchpad.net/bugs/1314686
>
> Title:
> Please add support for utopic
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1314686/+subscriptions
>

[Ian Booth (wallyworld)]

The fallback if there's no ubuntu.csv file is to look at a static map of known series. "utopic" isn't there yet. AFAIK, we should only get the error in this bug if distro-info-data pkg is not installed on the machine wanting to use the local provider

[James Page (james-page)]

Would it be possible to provide this via the 1.18.x stable series? Currently I can't get 1.18.x uploads into utopic as it uses the local provider for DEP-8 gating tests, which currently don't know about 'utopic' as a series.

[Robie Basak (racb)]

I'm thinking about any implications for future releases. Are we going to get this failing on every release, assuming that we won't know the next release's codename in advance again? If so, this seems bad from a distribution perspective to me, as the need to SRU fixes to juju-core may often be the highest just after release, and this will block us (or at least slow us down while we arrange an exception to the usual process).

Can I suggest a dep8 test in packaging to fake a "future" codename to make sure that juju can work in that circumstance (at least for the local provider)? I'll happily write this given some guidance - how does juju pick up the current release name? Do I just need to mangle /etc/lsb-release so that "lsb_release" gives me something different?

If we do expect this to fail in a similar way every release, should I file a separate bug to track this?

[Ian Booth (wallyworld)]

We do need the actual series name and version number in the code. The reason is that there's validation to ensure that simplestreams metadata for images and tools is correct, and we need to check that the series information encoded in the metadata is correct. We read the series info from /usr/share/distro-info/ubuntu.csv. However, we can't guarantee that the distro-info-data package is installed, so use hard coded fallbacks just in case. I guess a question I have is do we really care about running Juju 1.18 on Utopic given that by the time Utopic is released, there will be a matching Juju version also. We've got some other 1.18 bugs to fix so we are also able to easily add Utopic support anyway in this case, but in general, I wonder what our policy will be?

[Robie Basak (racb)]

On Mon, May 12, 2014 at 02:32:02AM -0000, Ian Booth wrote:
> We do need the actual series name and version number in the code. The
> reason is that there's validation to ensure that simplestreams metadata
> for images and tools is correct, and we need to check that the series
> information encoded in the metadata is correct. We read the series info
> from /usr/share/distro-info/ubuntu.csv. However, we can't guarantee that
> the distro-info-data package is installed, so use hard coded fallbacks
> just in case.

In my failure case, I did have distro-info available, but juju still
failed for me on Utopic. I haven't verified this though.

BTW, are we sure that /usr/share/distro-info/ubuntu.csv a published
interface, or should we be calling distro-info for the data instead?

Maybe we could have juju-core in archive depend on distro-info-data, and
then dep8 test just this path with a newer release? This is for the
specific case of installing juju from the archive as shipped with a
release. Then this would no longer be an issue for juju in the archive.
I understand that other cases have different needs (and this particular
bug isn't an issue in that case).

> just in case. I guess a question I have is do we really care about
> running Juju 1.18 on Utopic given that by the time Utopic is released,
> there will be a matching Juju version also. We've got some other 1.18
> bugs to fix so we are also able to easily add Utopic support anyway in
> this case, but in general, I wonder what our policy will be?

The problem is that we cannot land even an unrelated SRU to Trusty,
because the standard SRU process is to fix the development release
first, and Utopic is broken so I can't verify anything.

This prevents James from bumping the microrelease in Trusty, so we can't
fix anything else in Trusty's juju right now.

This also affects other packages. For example, I cannot land a
juju-quickstart fix for bug 1306537.

We could ask for an exception, but it seems to me that we shouldn't have
it broken this way by design in the first place, so I'm asking if
there's a way we can fix it that works for all cases instead.

[Robie Basak (racb)]

I've failed to get this working on Utopic. The existing dep8 test (modified just to add distro-info-data as an extra dependency) passes on Trusty, but fails on Utopic. I've also tried replacing the upstream tarball with one generated from 1.18 branch bzr revno 2291, and this also fails on Utopic.

Logs attached.

[Ian Booth (wallyworld)]

The original issue reported in the bug - the inability to generate tools for utopic - has been fixed. The logs identify a new, separate issue - the fact that mongo appears to have not started. We would need to see the mongo logs to determine why that is. I'm closing this bug again and a new bug for the new mongo issue will need to be opened.

[James Page (james-page)]

The original bug report was " Please add support for utopic " - the log just happens to be the first problem hit.

This is currently blocking SRU activity into trusty - until juju-core works on utopic, we can't push an SRU into trusty.

[James Page (james-page)]

May 19 14:57:18 hendrix mongod.37017[2472]: Mon May 19 14:57:18.170 [initandlisten] MongoDB starting : pid=2472 port=37017 dbpath=/home/jamespage/.juju/local/db 64-bit host=hendrix
May 19 14:57:18 hendrix mongod.37017[2472]: Mon May 19 14:57:18.170 [initandlisten] db version v2.4.9
May 19 14:57:18 hendrix mongod.37017[2472]: Mon May 19 14:57:18.170 [initandlisten] git version: nogitversion
May 19 14:57:18 hendrix mongod.37017[2472]: Mon May 19 14:57:18.170 [initandlisten] build info: Linux panlong 3.2.0-37-generic #58-Ubuntu SMP Thu Jan 24 15:28:10 UTC 2013 x86_64 BOOST_LIB_VERSION=1_55
May 19 14:57:18 hendrix mongod.37017[2472]: Mon May 19 14:57:18.170 [initandlisten] allocator: tcmalloc
May 19 14:57:18 hendrix mongod.37017[2472]: Mon May 19 14:57:18.170 [initandlisten] options: { auth: true, bind_ip: "0.0.0.0", dbpath: "/home/jamespage/.juju/local/db", noprealloc: true, port: 37017, smallfiles: true, sslOnNormalPorts: true, sslPEMKeyFile: "/home/jamespage/.juju/local/server.pem", sslPEMKeyPassword: "<password>", syslog: true }
May 19 14:57:18 hendrix mongod.37017[2472]: Mon May 19 14:57:18.170 [initandlisten] exception in initAndListen std::exception: boost::filesystem::status: Permission denied: "/home/jamespage/.juju/local/db", terminating
May 19 14:57:18 hendrix mongod.37017[2472]: Mon May 19 14:57:18.170
May 19 14:57:18 hendrix mongod.37017[2472]: dbexit:
May 19 14:57:18 hendrix mongod.37017[2472]:
May 19 14:57:18 hendrix mongod.37017[2472]: Mon May 19 14:57:18.170 [initandlisten] shutdown: going to close listening sockets...
May 19 14:57:18 hendrix mongod.37017[2472]: Mon May 19 14:57:18.170 [initandlisten] shutdown: going to flush diaglog...
May 19 14:57:18 hendrix mongod.37017[2472]: Mon May 19 14:57:18.170 [initandlisten] shutdown: going to close sockets...
May 19 14:57:18 hendrix mongod.37017[2472]: Mon May 19 14:57:18.171 [initandlisten] shutdown: waiting for fs preallocator...
May 19 14:57:18 hendrix mongod.37017[2472]: Mon May 19 14:57:18.171 [initandlisten] shutdown: lock for final commit...
May 19 14:57:18 hendrix mongod.37017[2472]: Mon May 19 14:57:18.171 [initandlisten] shutdown: final commit...
May 19 14:57:18 hendrix mongod.37017[2472]: Mon May 19 14:57:18.171 [initandlisten] shutdown: closing all files...
May 19 14:57:18 hendrix mongod.37017[2472]: Mon May 19 14:57:18.171 [initandlisten] closeAllFiles() finished
May 19 14:57:18 hendrix mongod.37017[2472]: Mon May 19 14:57:18.171
May 19 14:57:18 hendrix mongod.37017[2472]: dbexit: really exiting now
May 19 14:57:18 hendrix mongod.37017[2472]:

[Ian Booth (wallyworld)]

I've raised a new bug to track the mongo issue lp:1321025

[Robie Basak (racb)]

Stuck in utopic-proposed due to bug 1329295.

[Launchpad Janitor (janitor)]

This bug was fixed in the package juju-core - 1.18.4-0ubuntu1

---------------
juju-core (1.18.4-0ubuntu1) utopic; urgency=medium

  * New upstream release, including essential fixes for supporting Utopic and
    all future releases:
    - Support Utopic and future releases by using distro-info-data over
      hardcoded defaults (LP: #1314686).
    - Correctly use juju-mongodb on Trusty and all future releases, rather than
      hardcoding Trusty as special (LP: #1321025).
  * Depend on distro-info so that Juju can use its data in preference to
    hard-coded defaults (LP: #1325025).
  * d/tests/*:
    - Remove hardcoding of precise.
    - Use --upload-tools to reduce external dependencies on local and manual provider testing.
    - Add stderr logging.
    - New test for manual provider against localhost.
    - Mark tests breaks-testbed and isolation-machine so that they do not step on each other.
    - Add "future release" testing for both local and manual providers.
 -- Robie Basak <email address hidden> Fri, 06 Jun 2014 15:53:28 +0000