precise services cannot be deployed (again)

Bug #1423036 reported by Curtis Hovey
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Fix Released
Undecided
Unassigned
juju-core
Fix Released
High
Ian Booth
1.22
Fix Released
Critical
Ian Booth

Bug Description

Commit 4c05466 Merge pull request #1597 from wallyworld/enable-image-caching … was merged into 1.22, and again no precise service can be deployed.
    http://juju-ci.vapour.ws:8080/job/local-deploy-precise-amd64
    http://juju-ci.vapour.ws:8080/job/aws-deploy-precise-amd64

Like bug 1407699 which was fixed bug disabling we see cloud-init report that many things are not installed

Feb 18 01:25:37 ip-172-31-24-59 [CLOUDINIT] __init__.py[ERROR]: config handling of la ndscape, None, [] failed
Feb 18 01:25:37 ip-172-31-24-59 [CLOUDINIT] __init__.py[ERROR]: config handling of timezone, None, [] failed
Feb 18 01:25:37 ip-172-31-24-59 [CLOUDINIT] __init__.py[ERROR]: config handling of puppet, None, [] failed
Feb 18 01:25:37 ip-172-31-24-59 [CLOUDINIT] __init__.py[ERROR]: config handling of chef, None, [] failed
Feb 18 01:25:37 ip-172-31-24-59 [CLOUDINIT] __init__.py[ERROR]: config handling of salt-minion, None, [] failed
Feb 18 01:25:37 ip-172-31-24-59 [CLOUDINIT] __init__.py[ERROR]: config handling of mcollective, None, [] failed
Feb 18 01:25:37 ip-172-31-24-59 [CLOUDINIT] __init__.py[ERROR]: config handling of disable-ec2-metadata, None, [] failed
Feb 18 01:25:37 ip-172-31-24-59 [CLOUDINIT] __init__.py[ERROR]: config handling of runcmd, None, [] failed
Feb 18 01:25:37 ip-172-31-24-59 [CLOUDINIT] cloud-init-cfg[ERROR]: errors running cloud_config [config]: ['landscape', 'timezone', 'puppet', 'chef', 'salt-minion', 'mcollective', 'disable-ec2-metadata', 'runcmd', 'byobu']

Revision history for this message
Curtis Hovey (sinzui) wrote :
no longer affects: juju-core
Revision history for this message
Ian Booth (wallyworld) wrote :
Download full text (4.2 KiB)

The issue originally was that packaging metadata for debs in the precise cloud archive was incorrect. This has now been fixed (I looked at the cloud archive repo to be sure). I can deploy a precise AWS environment just fine (see below). Note the line:

Installing package: cloud-image-utils

This was what broken previously before the upstream packaging change.

I looked at the log from a failed CI run http://juju-ci.vapour.ws:8080/job/aws-deploy-precise-amd64/2304/
and see this line:

The following packages will be REMOVED:
  cloud-init cloud-initramfs-growroot cloud-utils

This implies that a stale mirror is being used for retrieving the cloud archive packages. The above is what happens when debs without the correct metadata are used.

Can we confirm that the issue is not due to a stale apt mirror?

juju bootstrap --show-log
2015-02-18 03:13:25 INFO juju.cmd supercommand.go:37 running juju [1.22-beta4-utopic-amd64 gc]
2015-02-18 03:13:25 INFO juju.provider.ec2 provider.go:35 opening environment "amazon"
2015-02-18 03:13:31 INFO juju.network network.go:114 setting prefer-ipv6 to false
Bootstrapping environment "amazon"
2015-02-18 03:13:37 INFO juju.environs.tools tools.go:86 reading tools with major.minor version 1.22
Starting new instance for initial state server
Launching instance
2015-02-18 03:14:21 INFO juju.provider.ec2 environ.go:509 "us-east-1a" is constrained, trying another availability zone
2015-02-18 03:14:23 INFO juju.provider.ec2 environ.go:509 "us-east-1b" is constrained, trying another availability zone
2015-02-18 03:14:26 INFO juju.provider.ec2 environ.go:525 started instance "i-3425a1ce" in "us-east-1c"
 - i-3425a1ce
2015-02-18 03:14:36 INFO juju.environs.bootstrap bootstrap.go:178 newest version: 1.22-beta3
2015-02-18 03:14:36 WARNING juju.environs.bootstrap bootstrap.go:201 failed to find 1.22-beta4 tools, will attempt to use 1.22-beta3
2015-02-18 03:14:36 INFO juju.environs.bootstrap bootstrap.go:206 picked bootstrap tools version: 1.22-beta3
Installing Juju agent on bootstrap instance
Waiting for address
Attempting to connect to 50.17.99.1:22
Attempting to connect to 10.73.158.126:22
Logging to /var/log/cloud-init-output.log on remote host
Installing add-apt-repository
Adding apt repository: deb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/cloud-tools main
Running apt-get update
Running apt-get upgrade
Installing package: curl
Installing package: cpu-checker
Installing package: bridge-utils
Installing package: rsyslog-gnutls
Installing package: cloud-image-utils
Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz <[https://juju-dist.s3.amazonaws.com/tools/devel/juju-1.22-beta3-precise-amd64.tgz]>
Bootstrapping Juju machine agent
Starting Juju machine agent (jujud-machine-0)
Bootstrap complete
2015-02-18 03:20:43 INFO juju.api api.go:502 new API addresses to cache [50.17.99.1:17070 10.73.158.126:17070]
2015-02-18 03:20:43 INFO juju.cmd supercommand.go:417 command finished

$ juju deploy mysql
Added charm "cs:precise/mysql-51" to the environment.

$ juju s...

Read more...

Revision history for this message
Curtis Hovey (sinzui) wrote :

This is the aws cloud-init-output.log

Revision history for this message
Curtis Hovey (sinzui) wrote :

This is the cloud-init-output log from joyent

Revision history for this message
Curtis Hovey (sinzui) wrote :

I think the problem is that the packages needed are not the default packages yet. I can see the old and new packages in
    http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/c/cloud-utils/
and Lp confirms that the old packages are not superseded.
    https://launchpad.net/ubuntu/trusty/amd64/cloud-image-utils
says the new cloud-image-utils is *proposed* not updates as cloud images require
The same is true for
    https://launchpad.net/ubuntu/trusty/amd64/cloud-utils

Revision history for this message
Ian Booth (wallyworld) wrote :

So I'm confused.

The log has this:

Ign http://ubuntu-cloud.archive.canonical.com precise-updates/cloud-tools/main

which gets data from here:

http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/precise-updates/cloud-tools/main/binary-amd64/Packages

which contains an old cloud-utils version as per below.

And yet this is up to date:

https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/tools

Outdated info from precise-updates/cloud-tools/main:

Package: cloud-guest-utils
Source: cloud-utils
Priority: extra
Section: admin
Installed-Size: 74
Maintainer: Scott Moser <email address hidden>
Architecture: all
Version: 0.27-0ubuntu9~ctools0
Recommends: gdisk
Replaces: cloud-utils (<< 0.27-0ubuntu3)
Depends: e2fsprogs (>= 1.4), util-linux (>= 2.17.2), python
Conflicts: cloud-utils (<< 0.27-0ubuntu3)
Filename: pool/main/c/cloud-utils/cloud-guest-utils_0.27-0ubuntu9~ctools0_all.deb
Size: 14922
MD5sum: 0b21367255325e83ac5b6aad4b756d76
SHA1: 822e5b9fc5aa682fe1eb715ca106d64c235eb37d
SHA256: 48691c1f0a32d3ea838c0bc23daf7c5c5d7349fee8f506a17627cfdf6b80ee24
Description: cloud guest utilities
Description-md5: fda0c5237d2eca0ed770cea9796a01fa

Revision history for this message
Curtis Hovey (sinzui) wrote :

Opps, nm, I inverted my read of
    https://launchpad.net/ubuntu/trusty/amd64/cloud-image-utils
and
   https://launchpad.net/ubuntu/trusty/amd64/cloud-utils
The required packages were copied *from* proposed to updated on the 14th.

But this is contradicted by logs that show the older tools from the same archive.
   Get:1 http://ubuntu-cloud.archive.canonical.com/ubuntu/ precise-updates/cloud-tools/main cloud-image-utils all 0.27-0ubuntu9~ctools0 [28.9 kB]

Revision history for this message
Curtis Hovey (sinzui) wrote :

We see Joyent, aws, and local-provider looking at precise-updates/cloud-tools
Get:8 http://ubuntu-cloud.archive.canonical.com precise-updates/cloud-tools Release
Get:11 http://ubuntu-cloud.archive.canonical.com precise-updates/cloud-tools/main amd64 Packages
Get:14 http://ubuntu-cloud.archive.canonical.com precise-updates/cloud-tools/main i386 Packages
and installing old cloud-image-utls

We see the old package is listed in the precise-updates
http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/precise-updates/cloud-tools/Release
http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/precise-updates/cloud-tools/main/binary-amd64/Packages
http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/precise-updates/cloud-tools/main/binary-i386/Packages

The new packages *are* listed in precise-proposed
http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/precise-proposed/cloud-tools/main/binary-amd64/Packages
http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/precise-proposed/cloud-tools/main/binary-i386/Packages

So it appears that contrary to Lp's record of publication, the packages are still in proposed on disk.

Revision history for this message
Robie Basak (racb) wrote :

Given the criticality of this issue, let's treat it is an SRU regression in cloud-init unless/until we determine otherwise.

tags: added: regression-update
Revision history for this message
Robie Basak (racb) wrote :

Oh - it's cloud archive specific, so not a general SRU issue.

tags: removed: regression-update
Robie Basak (racb)
affects: cloud-init (Ubuntu) → cloud-archive
Revision history for this message
Curtis Hovey (sinzui) wrote :

The server team confirmed that cloud-image-utils_0.27-0ubuntu9.1~ctools0_all is NOT in cloud-tools precise-updates. It is only proposed at this time. The change to juju was made prematurely. Our options are to revert, or to copy the package from proposed to updates.

Revision history for this message
Robie Basak (racb) wrote :

Steps to reproduce:

1. Start a fresh Precise VM (using an official released cloud image).
2. sudo add-apt-repository -y cloud-archive:tools
3. sudo apt-get update
4. sudo apt-get install cloud-image-utils

apt will want to:

The following extra packages will be installed:
  distro-info distro-info-data genisoimage libaio1 libnspr4 libnss3 librados2
  librbd1 python-distro-info qemu-utils
Suggested packages:
  shunit2 wodim cdrkit-doc
The following packages will be REMOVED:
  cloud-init cloud-initramfs-growroot cloud-utils
The following NEW packages will be installed:
  cloud-image-utils distro-info distro-info-data genisoimage libaio1 libnspr4
  libnss3 librados2 librbd1 python-distro-info qemu-utils

Expected behaviour: cloud-init should not be removed.

Juju may be doing something wrong here, but isn't it the case that we shouldn't really ever have the cloud tools pocket in a state where this might happen?

Revision history for this message
Robie Basak (racb) wrote :

Workaround: enable cloud-tools/precise-proposed.

Curtis Hovey (sinzui)
Changed in juju-core:
milestone: none → 1.23
status: New → Triaged
importance: Undecided → High
Curtis Hovey (sinzui)
Changed in cloud-archive:
status: New → Fix Released
Revision history for this message
Curtis Hovey (sinzui) wrote :

sign. We still have problems even after the new packages arrived in precise-updates/cloud-tools. My reading of the attached log is:

1. apt-get update was called and we see
    Get:6 http://ubuntu-cloud.archive.canonical.com precise-updates/cloud-tools/main amd64 Package
2, We can read that same file
    http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/precise-updates/cloud-tools/main/binary-amd64/Packages
    and it lists cloud-utils 0.27-0ubuntu9.1~ctools0
3. apt-get upgrade is called (and this had to be explicitly set for local-provider which wont do this by default)
    and we DO NOT see cloud-utils listed for upgrade
4. apt-get install is called and we see
    cloud-image-utils
    to be installed, BUT we see
    cloud-utils
    to be REMOVED.
5. We see cloud-image-utils is the correct version
   Get:1 http://ubuntu-cloud.archive.canonical.com/ubuntu/ precise-updates/cloud-tools/main cloud-image-utils all 0.27-0ubuntu9.1~ctools0
6. Everything ends in error because cloud-utils was removed.

Maybe we need to explicitly install cloud-utils to force an update of the version with the install of cloud-image-utils.

Precise cloud images do not use the cloud-tools archive. It uses standard precise updates (cloud-utils 0.25-0ubuntu5.1). We cannot expect new images to come with the better versions.

Revision history for this message
Curtis Hovey (sinzui) wrote :
Download full text (3.4 KiB)

I tested a precise cloud image to learn the apt-behaviour with the needed
packages. I confirmed precise images just use precise updates and has
very old-cloud-utils.

    apt-cache policy cloud-utils
        cloud-utils:
          Installed: 0.25-0ubuntu5.1
          Version table:
         *** 0.25-0ubuntu5.1 0
                500 http://joyent.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
                100 /var/lib/dpkg/status
             0.25-0ubuntu5 0
                500 http://joyent.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages

Adding cloud-archive:tools provides what we want, but update, upgrade will
not update cloud-utils. It is held back possibly because of conflicts.

    sudo add-apt-repository cloud-archive:tools
    sudo apt-get update
    apt-cache policy cloud-utils
        cloud-utils:
          Installed: 0.25-0ubuntu5.1
          Candidate: 0.27-0ubuntu9.1~ctools0
          Version table:
             0.27-0ubuntu9.1~ctools0 0
                500 http://ubuntu-cloud.archive.canonical.com/ubuntu/ precise-updates/cloud-tools/main amd64 Packages
         *** 0.25-0ubuntu5.1 0
                500 http://joyent.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
                100 /var/lib/dpkg/status
             0.25-0ubuntu5 0
                500 http://joyent.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages

    sudo apt-get upgrade
        Reading package lists... Done
        Building dependency tree
        Reading state information... Done
        The following packages have been kept back:
          cloud-utils iproute isc-dhcp-client isc-dhcp-common linux-headers-virtual
          linux-image-virtual linux-virtual
        The following packages will be upgraded:
          bind9-host dnsutils file krb5-locales landscape-client landscape-common
          libbind9-80 libdns81 libgssapi-krb5-2 libisc83 libisccc80 libisccfg82
          libk5crypto3 libkrb5-3 libkrb5support0 liblwres80 libmagic1 libudev0 mountall
          ntpdate procps python-apt python-apt-common python-twisted-bin
          python-twisted-core python-twisted-names python-twisted-web tzdata udev
          update-manager-core

But if we ask to install BOTH cloud-utils cloud-image-utils we get the right
version.

    sudo apt-get install cloud-utils cloud-image-utils
        Reading package lists... Done
        Building dependency tree
        Reading state information... Done
        The following extra packages will be installed:
          cloud-guest-utils distro-info distro-info-data gdisk genisoimage libaio1 libicu48
          libnspr4 libnss3 librados2 librbd1 python-distro-info qemu-utils
        Suggested packages:
          shunit2 wodim cdrkit-doc
        The following NEW packages will be installed:
          cloud-guest-utils cloud-image-utils distro-info distro-info-data gdisk
          genisoimage libaio1 libicu48 libnspr4 libnss3 librados2 librbd1
          python-distro-info qemu-utils
        The following packages will be upgraded:
          cloud-utils
        1 upgraded, 14 newly installed, 0 to remove and 36 not upgraded.

And asking for every package that I know we want in...

Read more...

Revision history for this message
Ian Booth (wallyworld) wrote :

I changed the cloud init script to do this for precise:

apt-get --target-release precise-updates/cloud-tools cloud-utils cloud-image-utils

This results in:

ubuntu@ip-10-73-159-15:~$ apt-cache policy cloud-utils
cloud-utils:
  Installed: 0.27-0ubuntu9.1~ctools0
  Candidate: 0.27-0ubuntu9.1~ctools0
  Version table:
 *** 0.27-0ubuntu9.1~ctools0 0
        400 http://ubuntu-cloud.archive.canonical.com/ubuntu/ precise-updates/cloud-tools/main amd64 Packages
        100 /var/lib/dpkg/status
     0.25-0ubuntu5.1 0
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
     0.25-0ubuntu5 0
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
ubuntu@ip-10-73-159-15:~$ apt-cache policy cloud-image-utils
cloud-image-utils:
  Installed: 0.27-0ubuntu9.1~ctools0
  Candidate: 0.27-0ubuntu9.1~ctools0
  Version table:
 *** 0.27-0ubuntu9.1~ctools0 0
        400 http://ubuntu-cloud.archive.canonical.com/ubuntu/ precise-updates/cloud-tools/main amd64 Packages
        100 /var/lib/dpkg/status

Changed in juju-core:
assignee: nobody → Ian Booth (wallyworld)
status: Triaged → In Progress
Revision history for this message
Ian Booth (wallyworld) wrote :

I tested on AWS with precise, trusty, utopic

Revision history for this message
Dimiter Naydenov (dimitern) wrote :

As Ian's patch https://github.com/juju/juju/pull/1637 has landed, marking this as Fix Committed.

Changed in juju-core:
status: In Progress → Fix Committed
Revision history for this message
Scott Moser (smoser) wrote :

fwiw, bug 1418568 is *just now* fixed for juju's use.
I uploaded the fix to cloud-tools -proposed a while ago and simutaneously uploaded that to the trusty ubuntu archive. I just now copied that fix from -proposed to -updates.

So, prior to today, jujuprobably shouldn't have seen this fix, other than in hacked developer versions that enable the 'cloud tools proposed' pocket. I hope you didn't/don't enable -proposed usage in juju stable.

Revision history for this message
Scott Moser (smoser) wrote :

ignore my nonsense comment 19.
this should be fixed now properly in the cloud tools archive in -updates, but the fix went in on the 19th.

Revision history for this message
Scott Moser (smoser) wrote :

ok,
I believe that the crux of this problem is in juju's use of 'apt-get upgrade' rather than 'apt-get dist-upgrade'.
'upgrade', per apt-get(8) :
    under no circumstances are currently installed packages removed, or packages not already installed retrieved and installed. New versions of currently installed packages that cannot be upgraded without changing the install status of another package will be left at their current version.

dist-upgrade will actually do the right thing.
  dist-upgrade in addition to performing the function of upgrade, also intelligently handles changing dependencies with new versions of packages

Curtis Hovey (sinzui)
Changed in juju-core:
status: Fix Committed → Fix Released
Curtis Hovey (sinzui)
Changed in juju-core:
milestone: 1.23 → 1.23-beta1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.