Canonical Juju

[RFE] lazy installation of agent dependencies

Bug #1773756 reported by Dmitrii Shcherbakov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Triaged
Low
Unassigned

Bug Description

Currently every machine used by Juju has a set of packages installed unconditionally.

Comparing cloud image manifest

https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64.manifest

with systems provisioned via Juju & MAAS I can see the following additional packages installed:

1) controller machines:
bridge-utils, cloud-image-utils, cloud-utils, cpu-checker, distro-info, genisoimage, juju-mongo-tools3.2, juju-mongodb3.2, libaio1:amd64, libboost-chrono1.58.0:amd64, libboost-filesystem1.58.0:amd64, libboost-iostreams1.58.0:amd64, libboost-program-options1.58.0:amd64, libboost-random1.58.0:amd64, libboost-regex1.58.0:amd64, libboost-system1.58.0:amd64, libboost-thread1.58.0:amd64, libgoogle-perftools4, libiscsi2:amd64, libnspr4:amd64, libnss3:amd64, libnss3-nssdb, libpcrecpp0v5:amd64, libplymouth4:amd64, librados2, librbd1, libsmartcols1:amd64, libsnappy1v5:amd64, libtcmalloc-minimal4, libunwind8, libyaml-cpp0.5v5:amd64, msr-tools, qemu-block-extra:amd64, qemu-utils, sharutils, ubuntu-fan

non-controller ubuntu machines (without charms):
bridge-utils, cloud-image-utils, cloud-utils, cpu-checker, distro-info, genisoimage, libaio1:amd64, libboost-iostreams1.58.0:amd64, libboost-random1.58.0:amd64, libboost-system1.58.0:amd64, libboost-thread1.58.0:amd64, libiscsi2:amd64, libnspr4:amd64, libnss3:amd64, libnss3-nssdb, librados2, librbd1, msr-tools, qemu-block-extra:amd64, qemu-utils, sharutils, ubuntu-fan

libboost, cpu-checker, msr-tools, libnss, ceph rbd-related dependencies and some others are for qemu-utils

Not every system we provision uses those packages as "kvm containers" are subjectively used less frequently than LXD containers in Juju models.

Those dependencies should be made optional for at least 2 reasons:

1) some users/customers ask to reduce the amount of packages installed to have a smaller attack surface;
2) installation speed.

Revision history for this message
John A Meinel (jameinel) wrote : Re: [Bug 1773756] [NEW] [RFE] lazy installation of agent dependencies

The one we actually want is 'cpu-checker' as we want to know whether we
*could* launch a KVM guest on a given machine, and we could defer
installing the rest of the packages to actually launch them until we
actually have requested it.
We could potentially have a flag for disabling KVM support entirely, it
doesn't really make sense to do it on a per-machine level.

On Mon, May 28, 2018 at 2:34 AM, Dmitrii Shcherbakov <
<email address hidden>> wrote:

> Public bug reported:
>
> Currently every machine used by Juju has a set of packages installed
> unconditionally.
>
> Comparing cloud image manifest
>
> https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-
> amd64.manifest
>
> with systems provisioned via Juju & MAAS I can see the following
> additional packages installed:
>
> 1) controller machines:
> bridge-utils, cloud-image-utils, cloud-utils, cpu-checker, distro-info,
> genisoimage, juju-mongo-tools3.2, juju-mongodb3.2, libaio1:amd64,
> libboost-chrono1.58.0:amd64, libboost-filesystem1.58.0:amd64,
> libboost-iostreams1.58.0:amd64, libboost-program-options1.58.0:amd64,
> libboost-random1.58.0:amd64, libboost-regex1.58.0:amd64,
> libboost-system1.58.0:amd64, libboost-thread1.58.0:amd64,
> libgoogle-perftools4, libiscsi2:amd64, libnspr4:amd64, libnss3:amd64,
> libnss3-nssdb, libpcrecpp0v5:amd64, libplymouth4:amd64, librados2, librbd1,
> libsmartcols1:amd64, libsnappy1v5:amd64, libtcmalloc-minimal4, libunwind8,
> libyaml-cpp0.5v5:amd64, msr-tools, qemu-block-extra:amd64, qemu-utils,
> sharutils, ubuntu-fan
>
> non-controller ubuntu machines (without charms):
> bridge-utils, cloud-image-utils, cloud-utils, cpu-checker, distro-info,
> genisoimage, libaio1:amd64, libboost-iostreams1.58.0:amd64,
> libboost-random1.58.0:amd64, libboost-system1.58.0:amd64,
> libboost-thread1.58.0:amd64, libiscsi2:amd64, libnspr4:amd64,
> libnss3:amd64, libnss3-nssdb, librados2, librbd1, msr-tools,
> qemu-block-extra:amd64, qemu-utils, sharutils, ubuntu-fan
>
> libboost, cpu-checker, msr-tools, libnss, ceph rbd-related dependencies
> and some others are for qemu-utils
>
> Not every system we provision uses those packages as "kvm containers"
> are subjectively used less frequently than LXD containers in Juju
> models.
>
> Those dependencies should be made optional for at least 2 reasons:
>
> 1) some users/customers ask to reduce the amount of packages installed to
> have a smaller attack surface;
> 2) installation speed.
>
> ** Affects: juju
> Importance: Undecided
> Status: New
>
>
> ** Tags: cpe-onsite
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1773756
>
> Title:
> [RFE] lazy installation of agent dependencies
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1773756/+subscriptions
>

John A Meinel (jameinel)
Changed in juju:
importance: Undecided → Medium
status: New → Triaged
Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

cpu-checker looks thin in terms of dependencies so installing that and deferring the rest seems good:

➜ ~ apt-rdepends cpu-checker
Reading package lists... Done
Building dependency tree
Reading state information... Done
cpu-checker
  Depends: msr-tools
msr-tools
  Depends: libc6 (>= 2.4)
libc6
  Depends: libgcc1
libgcc1
  Depends: gcc-8-base (= 8-20180414-1ubuntu2)
  Depends: libc6 (>= 2.14)
gcc-8-base
➜ ~ apt-rdepends msr-tools
Reading package lists... Done
Building dependency tree
Reading state information... Done
msr-tools
  Depends: libc6 (>= 2.4)
libc6
  Depends: libgcc1
libgcc1
  Depends: gcc-8-base (= 8-20180414-1ubuntu2)
  Depends: libc6 (>= 2.14)

Revision history for this message
John A Meinel (jameinel) wrote : Re: [Bug 1773756] Re: [RFE] lazy installation of agent dependencies

This is our list of packages:
func (cfg *ubuntuCloudConfig) addRequiredPackages() {
    packages := []string{
        "curl",
        "cpu-checker",
        // TODO(axw) 2014-07-02 #1277359
        // Don't install bridge-utils in cloud-init;
        // leave it to the networker worker.
        "bridge-utils",
        "cloud-utils",
        "tmux",
        // TODO(wpk) 2017-07-23 maybe we should do it in fanconfigurer?
        "ubuntu-fan",
    }

Things like qemu should only be installed if you actually request a KVM
instance.
Are you sure this isn't coming from somewhere else?

On Thu, Jun 7, 2018 at 11:40 AM, Dmitrii Shcherbakov <
<email address hidden>> wrote:

> cpu-checker looks thin in terms of dependencies so installing that and
> deferring the rest seems good:
>
> ➜ ~ apt-rdepends cpu-checker
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> cpu-checker
> Depends: msr-tools
> msr-tools
> Depends: libc6 (>= 2.4)
> libc6
> Depends: libgcc1
> libgcc1
> Depends: gcc-8-base (= 8-20180414-1ubuntu2)
> Depends: libc6 (>= 2.14)
> gcc-8-base
> ➜ ~ apt-rdepends msr-tools
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> msr-tools
> Depends: libc6 (>= 2.4)
> libc6
> Depends: libgcc1
> libgcc1
> Depends: gcc-8-base (= 8-20180414-1ubuntu2)
> Depends: libc6 (>= 2.14)
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1773756
>
> Title:
> [RFE] lazy installation of agent dependencies
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1773756/+subscriptions
>

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

Hmm, pretty sure.

$ juju bootstrap localhost

$ juju ssh -m controller 0 'dpkg -l | grep qemu'
ii qemu-block-extra:amd64 1:2.5+dfsg-5ubuntu10.29 amd64 extra block backend modules for qemu-system and qemu-utils
ii qemu-utils 1:2.5+dfsg-5ubuntu10.29 amd64 QEMU utilities

/var/log/apt/history.log:
http://paste.ubuntu.com/p/fvPjPT4dj8/
Start-Date: 2018-06-04 14:17:15
Commandline: apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet install cloud-utils
Requested-By: ubuntu (1000)
Install: libnss3-nssdb:amd64 (2:3.28.4-0ubuntu0.16.04.3, automatic), librbd1:amd64 (10.2.9-0ubuntu0.16.04.1, automatic), libboost-thread1.58.0:amd64 (1.58.0+dfsg-5ubuntu3.1, automatic), libboost-system1.58.0:amd64 (1.58.0+dfsg-5ubuntu3.1, automatic), libaio1:amd64 (0.3.110-2, automatic), libboost-random1.58.0:amd64 (1.58.0+dfsg-5ubuntu3.1, automatic), cloud-image-utils:amd64 (0.27-0ubuntu25.1, automatic), distro-info:amd64 (0.14build1, automatic), qemu-utils:amd64 (1:2.5+dfsg-5ubuntu10.29, automatic), sharutils:amd64 (1:4.15.2-1ubuntu0.1, automatic), genisoimage:amd64 (9:1.1.11-3ubuntu1, automatic), libiscsi2:amd64 (1.12.0-2, automatic), libnss3:amd64 (2:3.28.4-0ubuntu0.16.04.3, automatic), cloud-utils:amd64 (0.27-0ubuntu25.1), libboost-iostreams1.58.0:amd64 (1.58.0+dfsg-5ubuntu3.1, automatic), qemu-block-extra:amd64 (1:2.5+dfsg-5ubuntu10.29, automatic), librados2:amd64 (10.2.9-0ubuntu0.16.04.1, automatic), libnspr4:amd64 (2:4.13.1-0ubuntu0.16.04.1, automatic)
End-Date: 2018-06-04 14:17:42

Revision history for this message
John A Meinel (jameinel) wrote :

Doesn't that say that cloud-utils is requesting qemu-block-extra?

I think we do need at least some bits from cloud-utils but maybe we don't
need it as much anymore. (At one point we needed the package that helps
look up where cloud images would be found, but its possible that we don't
use it for that anymore.)
Similarly, I think we don't need bridge-utils installed by default if we
aren't creating bridges until late.

On Thu, Jun 7, 2018 at 1:28 PM, Dmitrii Shcherbakov <
<email address hidden>> wrote:

> Hmm, pretty sure.
>
> $ juju bootstrap localhost
>
> $ juju ssh -m controller 0 'dpkg -l | grep qemu'
> ii qemu-block-extra:amd64 1:2.5+dfsg-5ubuntu10.29
> amd64 extra block backend modules for qemu-system and
> qemu-utils
> ii qemu-utils 1:2.5+dfsg-5ubuntu10.29
> amd64 QEMU utilities
>
>
> /var/log/apt/history.log:
> http://paste.ubuntu.com/p/fvPjPT4dj8/
> Start-Date: 2018-06-04 14:17:15
> Commandline: apt-get --option=Dpkg::Options::=--force-confold
> --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet install
> cloud-utils
> Requested-By: ubuntu (1000)
> Install: libnss3-nssdb:amd64 (2:3.28.4-0ubuntu0.16.04.3, automatic),
> librbd1:amd64 (10.2.9-0ubuntu0.16.04.1, automatic),
> libboost-thread1.58.0:amd64 (1.58.0+dfsg-5ubuntu3.1, automatic),
> libboost-system1.58.0:amd64 (1.58.0+dfsg-5ubuntu3.1, automatic),
> libaio1:amd64 (0.3.110-2, automatic), libboost-random1.58.0:amd64
> (1.58.0+dfsg-5ubuntu3.1, automatic), cloud-image-utils:amd64
> (0.27-0ubuntu25.1, automatic), distro-info:amd64 (0.14build1, automatic),
> qemu-utils:amd64 (1:2.5+dfsg-5ubuntu10.29, automatic), sharutils:amd64
> (1:4.15.2-1ubuntu0.1, automatic), genisoimage:amd64 (9:1.1.11-3ubuntu1,
> automatic), libiscsi2:amd64 (1.12.0-2, automatic), libnss3:amd64
> (2:3.28.4-0ubuntu0.16.04.3, automatic), cloud-utils:amd64
> (0.27-0ubuntu25.1), libboost-iostreams1.58.0:amd64 (1.58.0+dfsg-5ubuntu3.1,
> automatic), qemu-block-extra:amd64 (1:2.5+dfsg-5ubuntu10.29, automatic),
> librados2:amd64 (10.2.9-0ubuntu0.16.04.1, automatic), libnspr4:amd64
> (2:4.13.1-0ubuntu0.16.04.1, automatic)
> End-Date: 2018-06-04 14:17:42
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1773756
>
> Title:
> [RFE] lazy installation of agent dependencies
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1773756/+subscriptions
>

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

bridge-utils is a very small package. I am more worried about cloud-utils which, as you mentioned, pulls in qemu-block-extra:

apt-rdepends cloud-utils | grep qemu
Reading package lists... Done
Building dependency tree
Reading state information... Done
  Depends: qemu-utils
qemu-utils
  Depends: qemu-block-extra (= 1:2.11+dfsg-1ubuntu7.3)
qemu-block-extra

Revision history for this message
Canonical Juju QA Bot (juju-qa-bot) wrote :

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.

Changed in juju:
importance: Medium → Low
tags: added: expirebugs-bot
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.