specifying apt_mirror of '' renders empty entries in /etc/apt/sources.list for uri

[Scott Moser]

==== Begin SRU Template ====
[Impact]
This is a bug that existed only in development time in cloud-init and was exposed during uploads to yakkety. It has never existed in xenial, but we do need to make sure that the function is still present.

The impact was that if a user (juju) started an instance with:
  apt_mirror: ''

that was taken as the apt mirror and /etc/apt/sources.list would have empty entries where a mirror should be.

[Test Case]
$ cat /tmp/foo.ud
#cloud-config
apt_mirror: ''

$ lxc launch ubuntu-daily:xenial sm-x0 "--config=user.user-data=$(cat /tmp/foo.ud)"

$ sleep 10

$ lxc exec sm-x0 grep yakkety /etc/apt/sources.list | head -n 3
deb yakkety main restricted
deb-src yakkety main restricted
deb yakkety-updates main restricted

Expected behavior is to see http://archive.ubuntu.com/ubuntu.

[Regression Potential]
Over all, the apt configuration feature could have other regressions
similar to this, but this specific change has very little chance for regresion.

==== End SRU Template ====

$ cat /tmp/foo.ud
#cloud-config
apt_mirror: ''

$ lxc launch ubuntu-daily:yakkety sm-y0 "--config=user.user-data=$(cat /tmp/foo.ud)"

$ sleep 10

$ lxc exec sm-y0 grep ^deb /etc/apt/sources.list | head -n 3
deb yakkety main restricted
deb-src yakkety main restricted
deb yakkety-updates main restricted

## Now get -proposed version.
$ lxc exec sm-x0 -- sh -c '
    p=/etc/apt/sources.list.d/proposed.list
    echo deb http://archive.ubuntu.com/ubuntu xenial-proposed main > "$p" &&
    apt-get update -q && apt-get -qy install cloud-init'

## cleanup so it thinks its fresh
$ lxc exec sm-x0 -- sh -c '
    cd /var/lib/cloud && for d in *; do [ "$d" = "seed" ] || rm -Rf "$d"; done
    rm -Rf /var/log/cloud-init*'

$ lxc exec sm-x0 reboot
$ lxc exec sm-x0 grep ^deb /etc/apt/sources.list | head -n 3
deb http://archive.ubuntu.com/ubuntu xenial main restricted
deb-src http://archive.ubuntu.com/ubuntu xenial main restricted
deb http://archive.ubuntu.com/ubuntu xenial-updates main restricted

basically if you provide an empty apt_mirror in the old format, then it is taken as providing an apt mirror. This non-true value should just be the same as not providing it.

ProblemType: Bug
DistroRelease: Ubuntu 16.10
Package: cloud-init 0.7.7-22-g763f403-0ubuntu1
ProcVersionSignature: Ubuntu 4.4.0-9136.55-generic 4.4.16
Uname: Linux 4.4.0-9136-generic x86_64
ApportVersion: 2.20.3-0ubuntu7
Architecture: amd64
Date: Wed Sep 7 17:12:11 2016
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
SourcePackage: cloud-init
UpgradeStatus: No upgrade log present (probably fresh install)

[Scott Moser (smoser)]

I made this affect juju also, as where this bug was filed juju explicitly declared:

$ sudo grep apt_mirror /var/lib/cloud/instance/cloud-config.txt
apt_mirror: ''

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.7-26-g058dd75-0ubuntu1

---------------
cloud-init (0.7.7-26-g058dd75-0ubuntu1) yakkety; urgency=medium

  * New upstream snapshot.
    - apt config conversion: treat empty string as not provided. (LP: #1621180)
    - Fix typo in default keys for phone_home [Roland Sommer] (LP: #1607810)
    - salt minion: update default pki directory for newer salt minion.
      (LP: #1609899)
    - bddeb: add --release flag to specify the release in changelog.

 -- Scott Moser <email address hidden> Thu, 08 Sep 2016 09:36:52 -0400

[Scott Moser (smoser)]

fixed in 0.7.8.

[Chris J Arges (arges)]

Hello Scott, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7-31-g65ace7b-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

Verified with:

lxc launch ubuntu-daily:xenial sm-x0 "--config=user.user-data=$(cat /tmp/foo.ud)"
## get proposed version
lxc exec sm-x0 -- sh -c '
    p=/etc/apt/sources.list.d/proposed.list
    echo deb http://archive.ubuntu.com/ubuntu xenial-proposed main > "$p" &&
    apt-get update -q && apt-get -qy install cloud-init'

## cleanup
lxc exec sm-x0 -- sh -c '
    cd /var/lib/cloud && for d in *; do [ "$d" = "seed" ] || rm -Rf "$d"; done
    rm -Rf /var/log/cloud-init*'

lxc exec sm-x0 reboot
$ lxc exec sm-x0 grep ^deb /etc/apt/sources.list | head -n 3
deb http://archive.ubuntu.com/ubuntu xenial main restricted
deb-src http://archive.ubuntu.com/ubuntu xenial main restricted
deb http://archive.ubuntu.com/ubuntu xenial-updates main restricted

[Martin Pitt (pitti)]

Hello Scott, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.8-1-g3705bb5-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

verified as shown in template.
$ printf "#cloud-config\napt_mirror: ''\n" > user-data
$ n=x1
$ lxc delete --force $n
$ lxc launch ubuntu-daily:xenial $n
$ sleep 10
$ lxc exec $n -- sh -c '
    p=/etc/apt/sources.list.d/proposed.list
    echo deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-proposed main > "$p" &&
    apt-get update -q && apt-get -qy install cloud-init' </dev/null

$ lxc file push - $n/etc/cloud/cloud.cfg.d/mirror.cfg < user-data

## clean it out so next is first boot.
$ lxc exec $n -- sh -c '
  cd /var/lib/cloud && for d in *; do [ "$d" = "seed" ] || rm -Rf "$d"; done
  rm -Rf /var/log/cloud-init*'

$ lxc exec $n reboot
$ lxc exec $n -- tail -f /var/log/cloud-init-output.log
$ lxc exec $n grep ^deb /etc/apt/sources.list | head -n 3
deb http://archive.ubuntu.com/ubuntu xenial main restricted
deb-src http://archive.ubuntu.com/ubuntu xenial main restricted
deb http://archive.ubuntu.com/ubuntu xenial-updates main restricted

$ lxc exec $n -- dpkg-query --show cloud-init
cloud-init 0.7.8-1-g3705bb5-0ubuntu1~16.04.1

[Scott Moser (smoser)]

## Recreate failure
$ cat >user-data <<EOF
#cloud-config
salt_minion:
  public_key: "foo public"
  private_key: "foo private"
EOF

$ lxc delete --force $n
$ lxc launch ubuntu-daily:xenial $n "--config=user.user-data=$(cat user-data)"
sleep 30
$ lxc exec $n -- grep salt/pki/ /var/log/cloud-init.log </dev/null
$ lxc exec $n -- grep salt/pki/ /var/log/cloud-init.log </dev/null
Sep 17 02:30:03 ubuntu [CLOUDINIT] util.py[DEBUG]: Writing to /etc/salt/pki/minion.pub - wb: [420] 10 bytes
Sep 17 02:30:03 ubuntu [CLOUDINIT] util.py[DEBUG]: Writing to /etc/salt/pki/minion.pem - wb: [420] 11 bytes

## Note, that ubuntu's packaging actuall moves these files to their proper
## location, so checking the log is all we can do to show failure.

## Now update container, clean and reboot to show first boot
$ lxc exec $n -- sh -c '
    p=/etc/apt/sources.list.d/proposed.list
    echo deb http://archive.ubuntu.com/ubuntu xenial-proposed main > "$p" &&
    apt-get update -q && apt-get -qy install cloud-init' </dev/null
$ lxc exec $n -- sh -c 'apt-get -qy --purge remove salt-minion && rm -Rf /etc/salt' </dev/null
$ lxc exec $n -- sh -c '
    cd /var/lib/cloud && for d in *; do [ "$d" = "seed" ] || rm -Rf "$d"; done
    rm -Rf /var/log/cloud-init*'

$ lxc exec $n reboot

$ lxc exec $n -- grep salt/pki/ /var/log/cloud-init.log
Sep 17 02:31:17 x1 [CLOUDINIT] util.py[DEBUG]: Writing to /etc/salt/pki/minion/minion.pub - wb: [420] 10 bytes
Sep 17 02:31:17 x1 [CLOUDINIT] util.py[DEBUG]: Writing to /etc/salt/pki/minion/minion.pem - wb: [420] 11 bytes

$ lxc exec $n -- dpkg-query --show cloud-init
cloud-init 0.7.8-1-g3705bb5-0ubuntu1~16.04.1

[Scott Moser (smoser)]

verified. comment above was on wrong bug, but verified this one as well.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.8-1-g3705bb5-0ubuntu1~16.04.1

---------------
cloud-init (0.7.8-1-g3705bb5-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * New upstream release 0.7.8.
  * New upstream snapshot.
    - systemd: put cloud-init.target After multi-user.target (LP: #1623868)

cloud-init (0.7.7-31-g65ace7b-0ubuntu1~16.04.2) xenial-proposed; urgency=medium

  * debian/control: add Breaks of older versions of walinuxagent (LP: #1623570)

cloud-init (0.7.7-31-g65ace7b-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/control: fix missing dependency on python3-serial,
    and make SmartOS datasource work.
  * debian/cloud-init.templates fix capitalisation in template so
    dpkg-reconfigure works to select OpenStack. (LP: #1575727)
  * d/README.source, d/control, d/new-upstream-snapshot, d/rules: sync
    with yakkety for changes due to move to git.
  * d/rules: change PYVER=python3 to PYVER=3 to adjust to upstream change.
  * debian/rules, debian/cloud-init.install: remove install file
    to ensure expected files are collected into cloud-init deb.
    (LP: #1615745)
  * debian/dirs: remove obsolete / unused file.
  * upstream move from bzr to git.
  * New upstream snapshot.
    - Allow link type of null in network_data.json [Jon Grimm] (LP: #1621968)
    - DataSourceOVF: fix user-data as base64 with python3 (LP: #1619394)
    - remove obsolete .bzrignore
    - systemd: Better support package and upgrade. (LP: #1576692, #1621336)
    - tests: cleanup tempdirs in apt_source tests
    - apt config conversion: treat empty string as not provided. (LP: #1621180)
    - Fix typo in default keys for phone_home [Roland Sommer] (LP: #1607810)
    - salt minion: update default pki directory for newer salt minion.
      (LP: #1609899)
    - bddeb: add --release flag to specify the release in changelog.
    - apt-config: allow both old and new format to be present.
      [Christian Ehrhardt] (LP: #1616831)
    - python2.6: fix dict comprehension usage in _lsb_release. [Joshua Harlow]
    - Add a module that can configure spacewalk. [Joshua Harlow]
    - add install option for openrc [Matthew Thode]
    - Generate a dummy bond name for OpenStack (LP: #1605749)
    - network: fix get_interface_mac for bond slave, read_sys_net for ENOTDIR
    - azure dhclient-hook cleanups
    - Minor cleanups to atomic_helper and add unit tests.
    - Fix Gentoo net config generation [Matthew Thode]
    - distros: fix get_primary_arch method use of os.uname [Andrew Jorgensen]
    - Apt: add new apt configuration format [Christian Ehrhardt]
    - Get Azure endpoint server from DHCP client [Brent Baude]
    - DigitalOcean: use the v1.json endpoint [Ben Howard]
    - MAAS: add vendor-data support (LP: #1612313)
    - Upgrade to a configobj package new enough to work [Joshua Harlow]
    - ConfigDrive: recognize 'tap' as a link type. (LP: #1610784)
    - NoCloud: fix bug providing network-interfaces via meta-data.
      (LP: 1577982)
    - Add distro tags on config modules that should have it [Joshua Harlow]
    - ChangeLog: update changelog for previous commit.
    - add ntp config module [Ryan Harper]
    - SmartOS: more improvement...

[Chris J Arges (arges)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2723