rabbitmq-server package got upgraded after system restart

[Yoshi Kadokawa]

[Deployed environment]
Juju: v2.1.2
charm: cs:rabbitmq-server-61
OS: Ubuntu 16.04

[Description]
After system reboot, it looks like that rabbitmq-server package got upgraded during the config-changed hook.
I could see that the upgrade was done in the charm, according to this log from /var/log/juju/unit-rabbitmq-server-2.log

2017-05-11 13:30:12 INFO juju-log Installing ['rabbitmq-server', 'python-amqplib', 'lockfile-progs'] with options: ['--option=Dpkg::Options::=--force-confold']
2017-05-11 13:30:12 INFO config-changed Reading package lists...
2017-05-11 13:30:12 INFO config-changed Building dependency tree...
2017-05-11 13:30:12 INFO config-changed Reading state information...
2017-05-11 13:30:12 INFO config-changed lockfile-progs is already the newest version (0.1.17).
2017-05-11 13:30:12 INFO config-changed python-amqplib is already the newest version (1.0.2-1).
2017-05-11 13:30:12 INFO config-changed The following packages will be upgraded:
2017-05-11 13:30:12 INFO config-changed rabbitmq-server
2017-05-11 13:30:13 INFO config-changed 1 upgraded, 0 newly installed, 0 to remove and 35 not upgraded.

According to the hook script[0], it looks like apt_install() function is being called in config-changed hook.
Therefore, by restarting the jujud-unit-rabbitmq-server-x, config-changed hook got triggered, and the rabbitmq-server got upgraded.
I think this behavior is not necessary, and should not be triggered in config-changed hook.

[Reproduction steps]
I could reproduce this with the following steps.

$ juju deploy rabbitmq-server
$ juju ssh rabbitmq-server/0
$ apt policy rabbitmq-server
rabbitmq-server:
Installed: 3.5.7-1ubuntu0.16.04.1
Candidate: 3.5.7-1ubuntu0.16.04.1
Version table:
*** 3.5.7-1ubuntu0.16.04.1 500
500 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
100 /var/lib/dpkg/status
3.5.7-1 500
500 http://nova.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
$ sudo apt install rabbitmq-server=3.5.7-1
$ apt policy rabbitmq-server
rabbitmq-server:
Installed: 3.5.7-1
Candidate: 3.5.7-1ubuntu0.16.04.1
Version table:
3.5.7-1ubuntu0.16.04.1 500
500 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
*** 3.5.7-1 500
500 http://nova.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status
$ sudo systemctl restart jujud-unit-rabbitmq-server-0.service
$ apt policy rabbitmq-server
rabbitmq-server:
Installed: 3.5.7-1ubuntu0.16.04.1
Candidate: 3.5.7-1ubuntu0.16.04.1
Version table:
*** 3.5.7-1ubuntu0.16.04.1 500
500 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
100 /var/lib/dpkg/status
3.5.7-1 500
500 http://nova.clouds.a$ apt policy percona-xtradb-cluster-server-5.6

[0] https://github.com/openstack/charm-rabbitmq-server/blob/master/hooks/rabbitmq_server_relations.py#L619

[Mario Splivalo (mariosplivalo)]

This also seems, at least, odd to me. It looks like the rationale was that the rabbitmq-server can be upgraded to version from a different source pocket (like, 'newer' cloud archive repos, or some other ppa) by merely doing 'juju set rabbitmq-server source="cloud:..." - that will change the sources.list and run apt-get update/upgrade mantra.

However, this has a undesired side effect: apt-get update/upgrade is being run each time config-hook is run - when machine gets rebooted, when state server(s) get rebooted, or when some other charm option has been changed and it means that the operator lost control on when the packages are being upgraded.

Usually the operator wants to control when packages are being upgraded; it is highly unusual that one would get packages upgraded after machine gets rebooted.

I've also checked other openstack charms and none of them exhibit this behavior. For all of the other charms setting the 'source' or 'origin' charm option will not auto-trigger apt-get update/upgrade mantra; this seems like the correct behavior. If an operator wishes to change/upgrade his or hers packages a 'juju set' accompanied with 'juju run' or desired action should be executed.

To be consistent with other charms (at least), the commit https://github.com/openstack/charm-rabbitmq-server/commit/28067861bf8711ec1359a87feebeb534a88028a7 (which introduced this change) should be, imho, reverted.

[Mario Splivalo (mariosplivalo)]

Based on discussion with Openstack Charmers, this is a bug, but not in a way I described.

The charm should check if 'source' changed (comparing to what is configured in sources.list), and only if there is a change it should trigger the apt-get update/upgarde mantra.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/479440

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/479440
Committed: https://git.openstack.org/cgit/openstack/charm-rabbitmq-server/commit/?id=9d9c6c778d0c286802194e2cba4b861450d17920
Submitter: Jenkins
Branch: master

commit 9d9c6c778d0c286802194e2cba4b861450d17920
Author: Mario Splivalo <email address hidden>
Date: Sat Jul 1 00:30:13 2017 +0200

    Prevent running apt-get update/upgrade on restart

    As config-changed hook runs each time jujud is restarted, the hook
    could upgrade rabbitmq without operator warrant.

    This change runs apt-get update/upgrade in config-changed only
    if "source" charm options is changed.

    Change-Id: I56781659b02b883c9b15867dac6f9c8d27b5d975
    Closes-Bug: 1694377