precise services cannot be deployed (again)
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| | Ubuntu Cloud Archive |
Undecided
|
Unassigned | ||
| | juju-core |
High
|
Ian Booth | ||
| | 1.22 |
Critical
|
Ian Booth | ||
Bug Description
Commit 4c05466 Merge pull request #1597 from wallyworld/
http://
http://
Like bug 1407699 which was fixed bug disabling we see cloud-init report that many things are not installed
Feb 18 01:25:37 ip-172-31-24-59 [CLOUDINIT] __init__.py[ERROR]: config handling of la ndscape, None, [] failed
Feb 18 01:25:37 ip-172-31-24-59 [CLOUDINIT] __init__.py[ERROR]: config handling of timezone, None, [] failed
Feb 18 01:25:37 ip-172-31-24-59 [CLOUDINIT] __init__.py[ERROR]: config handling of puppet, None, [] failed
Feb 18 01:25:37 ip-172-31-24-59 [CLOUDINIT] __init__.py[ERROR]: config handling of chef, None, [] failed
Feb 18 01:25:37 ip-172-31-24-59 [CLOUDINIT] __init__.py[ERROR]: config handling of salt-minion, None, [] failed
Feb 18 01:25:37 ip-172-31-24-59 [CLOUDINIT] __init__.py[ERROR]: config handling of mcollective, None, [] failed
Feb 18 01:25:37 ip-172-31-24-59 [CLOUDINIT] __init__.py[ERROR]: config handling of disable-
Feb 18 01:25:37 ip-172-31-24-59 [CLOUDINIT] __init__.py[ERROR]: config handling of runcmd, None, [] failed
Feb 18 01:25:37 ip-172-31-24-59 [CLOUDINIT] cloud-init-
| Ian Booth (wallyworld) wrote : | #2 |
| Curtis Hovey (sinzui) wrote : | #3 |
This is the aws cloud-init-
| Curtis Hovey (sinzui) wrote : | #4 |
This is the cloud-init-output log from joyent
| Curtis Hovey (sinzui) wrote : | #5 |
I think the problem is that the packages needed are not the default packages yet. I can see the old and new packages in
http://
and Lp confirms that the old packages are not superseded.
https:/
says the new cloud-image-utils is *proposed* not updates as cloud images require
The same is true for
https:/
| Ian Booth (wallyworld) wrote : | #6 |
So I'm confused.
The log has this:
Ign http://
which gets data from here:
which contains an old cloud-utils version as per below.
And yet this is up to date:
https:/
Outdated info from precise-
Package: cloud-guest-utils
Source: cloud-utils
Priority: extra
Section: admin
Installed-Size: 74
Maintainer: Scott Moser <email address hidden>
Architecture: all
Version: 0.27-0ubuntu9~
Recommends: gdisk
Replaces: cloud-utils (<< 0.27-0ubuntu3)
Depends: e2fsprogs (>= 1.4), util-linux (>= 2.17.2), python
Conflicts: cloud-utils (<< 0.27-0ubuntu3)
Filename: pool/main/
Size: 14922
MD5sum: 0b21367255325e8
SHA1: 822e5b9fc5aa682
SHA256: 48691c1f0a32d3e
Description: cloud guest utilities
Description-md5: fda0c5237d2eca0
| Curtis Hovey (sinzui) wrote : | #7 |
Opps, nm, I inverted my read of
https:/
and
https:/
The required packages were copied *from* proposed to updated on the 14th.
But this is contradicted by logs that show the older tools from the same archive.
Get:1 http://
| Curtis Hovey (sinzui) wrote : | #8 |
We see Joyent, aws, and local-provider looking at precise-
Get:8 http://
Get:11 http://
Get:14 http://
and installing old cloud-image-utls
We see the old package is listed in the precise-updates
http://
http://
http://
The new packages *are* listed in precise-proposed
http://
http://
So it appears that contrary to Lp's record of publication, the packages are still in proposed on disk.
| Robie Basak (racb) wrote : | #9 |
Given the criticality of this issue, let's treat it is an SRU regression in cloud-init unless/until we determine otherwise.
| tags: | added: regression-update |
| Robie Basak (racb) wrote : | #10 |
Oh - it's cloud archive specific, so not a general SRU issue.
| tags: | removed: regression-update |
| affects: | cloud-init (Ubuntu) → cloud-archive |
| Curtis Hovey (sinzui) wrote : | #11 |
The server team confirmed that cloud-image-
| Robie Basak (racb) wrote : | #12 |
Steps to reproduce:
1. Start a fresh Precise VM (using an official released cloud image).
2. sudo add-apt-repository -y cloud-archive:tools
3. sudo apt-get update
4. sudo apt-get install cloud-image-utils
apt will want to:
The following extra packages will be installed:
distro-info distro-info-data genisoimage libaio1 libnspr4 libnss3 librados2
librbd1 python-distro-info qemu-utils
Suggested packages:
shunit2 wodim cdrkit-doc
The following packages will be REMOVED:
cloud-init cloud-initramfs
The following NEW packages will be installed:
cloud-image-utils distro-info distro-info-data genisoimage libaio1 libnspr4
libnss3 librados2 librbd1 python-distro-info qemu-utils
Expected behaviour: cloud-init should not be removed.
Juju may be doing something wrong here, but isn't it the case that we shouldn't really ever have the cloud tools pocket in a state where this might happen?
| Robie Basak (racb) wrote : | #13 |
Workaround: enable cloud-tools/
| Changed in juju-core: | |
| milestone: | none → 1.23 |
| status: | New → Triaged |
| importance: | Undecided → High |
| Changed in cloud-archive: | |
| status: | New → Fix Released |
| Curtis Hovey (sinzui) wrote : | #14 |
sign. We still have problems even after the new packages arrived in precise-
1. apt-get update was called and we see
Get:6 http://
2, We can read that same file
http://
and it lists cloud-utils 0.27-0ubuntu9.
3. apt-get upgrade is called (and this had to be explicitly set for local-provider which wont do this by default)
and we DO NOT see cloud-utils listed for upgrade
4. apt-get install is called and we see
cloud-
to be installed, BUT we see
cloud-utils
to be REMOVED.
5. We see cloud-image-utils is the correct version
Get:1 http://
6. Everything ends in error because cloud-utils was removed.
Maybe we need to explicitly install cloud-utils to force an update of the version with the install of cloud-image-utils.
Precise cloud images do not use the cloud-tools archive. It uses standard precise updates (cloud-utils 0.25-0ubuntu5.1). We cannot expect new images to come with the better versions.
| Curtis Hovey (sinzui) wrote : | #15 |
I tested a precise cloud image to learn the apt-behaviour with the needed
packages. I confirmed precise images just use precise updates and has
very old-cloud-utils.
apt-cache policy cloud-utils
Version table:
*** 0.25-0ubuntu5.1 0
500 http://
100 /var/lib/
500 http://
Adding cloud-archive:tools provides what we want, but update, upgrade will
not update cloud-utils. It is held back possibly because of conflicts.
sudo add-apt-repository cloud-archive:tools
sudo apt-get update
apt-cache policy cloud-utils
Version table:
500 http://
*** 0.25-0ubuntu5.1 0
500 http://
100 /var/lib/
500 http://
sudo apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages have been kept back:
The following packages will be upgraded:
ntpdate procps python-apt python-apt-common python-twisted-bin
But if we ask to install BOTH cloud-utils cloud-image-utils we get the right
version.
sudo apt-get install cloud-utils cloud-image-utils
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
libnspr4 libnss3 librados2 librbd1 python-distro-info qemu-utils
Suggested packages:
shunit2 wodim cdrkit-doc
The following NEW packages will be installed:
The following packages will be upgraded:
1 upgraded, 14 newly installed, 0 to remove and 36 not upgraded.
And asking for every package that I know we want in...
| Ian Booth (wallyworld) wrote : | #16 |
I changed the cloud init script to do this for precise:
apt-get --target-release precise-
This results in:
ubuntu@
cloud-utils:
Installed: 0.27-0ubuntu9.
Candidate: 0.27-0ubuntu9.
Version table:
*** 0.27-0ubuntu9.
400 http://
100 /var/lib/
0.
500 http://
0.25-0ubuntu5 0
500 http://
ubuntu@
cloud-image-utils:
Installed: 0.27-0ubuntu9.
Candidate: 0.27-0ubuntu9.
Version table:
*** 0.27-0ubuntu9.
400 http://
100 /var/lib/
| Changed in juju-core: | |
| assignee: | nobody → Ian Booth (wallyworld) |
| status: | Triaged → In Progress |
| Ian Booth (wallyworld) wrote : | #17 |
I tested on AWS with precise, trusty, utopic
| Dimiter Naydenov (dimitern) wrote : | #18 |
As Ian's patch https:/
| Changed in juju-core: | |
| status: | In Progress → Fix Committed |
| Scott Moser (smoser) wrote : | #19 |
fwiw, bug 1418568 is *just now* fixed for juju's use.
I uploaded the fix to cloud-tools -proposed a while ago and simutaneously uploaded that to the trusty ubuntu archive. I just now copied that fix from -proposed to -updates.
So, prior to today, jujuprobably shouldn't have seen this fix, other than in hacked developer versions that enable the 'cloud tools proposed' pocket. I hope you didn't/don't enable -proposed usage in juju stable.
| Scott Moser (smoser) wrote : | #20 |
ignore my nonsense comment 19.
this should be fixed now properly in the cloud tools archive in -updates, but the fix went in on the 19th.
| Scott Moser (smoser) wrote : | #21 |
ok,
I believe that the crux of this problem is in juju's use of 'apt-get upgrade' rather than 'apt-get dist-upgrade'.
'upgrade', per apt-get(8) :
under no circumstances are currently installed packages removed, or packages not already installed retrieved and installed. New versions of currently installed packages that cannot be upgraded without changing the install status of another package will be left at their current version.
dist-upgrade will actually do the right thing.
dist-upgrade in addition to performing the function of upgrade, also intelligently handles changing dependencies with new versions of packages
| Changed in juju-core: | |
| status: | Fix Committed → Fix Released |
| Changed in juju-core: | |
| milestone: | 1.23 → 1.23-beta1 |
The issue originally was that packaging metadata for debs in the precise cloud archive was incorrect. This has now been fixed (I looked at the cloud archive repo to be sure). I can deploy a precise AWS environment just fine (see below). Note the line:
Installing package: cloud-image-utils
This was what broken previously before the upstream packaging change.
I looked at the log from a failed CI run http://
and see this line:
The following packages will be REMOVED:
cloud-init cloud-initramfs
This implies that a stale mirror is being used for retrieving the cloud archive packages. The above is what happens when debs without the correct metadata are used.
Can we confirm that the issue is not due to a stale apt mirror?
juju bootstrap --show-log
2015-02-18 03:13:25 INFO juju.cmd supercommand.go:37 running juju [1.22-beta4-
2015-02-18 03:13:25 INFO juju.provider.ec2 provider.go:35 opening environment "amazon"
2015-02-18 03:13:31 INFO juju.network network.go:114 setting prefer-ipv6 to false
Bootstrapping environment "amazon"
2015-02-18 03:13:37 INFO juju.environs.tools tools.go:86 reading tools with major.minor version 1.22
Starting new instance for initial state server
Launching instance
2015-02-18 03:14:21 INFO juju.provider.ec2 environ.go:509 "us-east-1a" is constrained, trying another availability zone
2015-02-18 03:14:23 INFO juju.provider.ec2 environ.go:509 "us-east-1b" is constrained, trying another availability zone
2015-02-18 03:14:26 INFO juju.provider.ec2 environ.go:525 started instance "i-3425a1ce" in "us-east-1c"
- i-3425a1ce
2015-02-18 03:14:36 INFO juju.environs.
2015-02-18 03:14:36 WARNING juju.environs.
2015-02-18 03:14:36 INFO juju.environs.
Installing Juju agent on bootstrap instance
Waiting for address
Attempting to connect to 50.17.99.1:22
Attempting to connect to 10.73.158.126:22
Logging to /var/log/
Installing add-apt-repository
Adding apt repository: deb http://
Running apt-get update
Running apt-get upgrade
Installing package: curl
Installing package: cpu-checker
Installing package: bridge-utils
Installing package: rsyslog-gnutls
Installing package: cloud-image-utils
Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz <[https:/
Bootstrapping Juju machine agent
Starting Juju machine agent (jujud-machine-0)
Bootstrap complete
2015-02-18 03:20:43 INFO juju.api api.go:502 new API addresses to cache [50.17.99.1:17070 10.73.158.
2015-02-18 03:20:43 INFO juju.cmd supercommand.go:417 command finished
$ juju deploy mysql
Added charm "cs:precise/
$ juju s...