MAAS uninstalls itself on unattended upgrade

[Chris Sanders]

Having recently found my MAAS unresponsive I was surprised to find out it's because it uninstalled itself. First, how on earth does this happen, and second how do I go about recovering this I have several critical juju services deployed on a nodes that this MAAS runs and juju is unusable with MAAS down.

I've just done the backup as per https://docs.maas.io/2.5/en/manage-backup do I just do a fresh install now and do the restore steps? Will this simply get removed on the next upgrade?

Start-Date: 2019-04-24 06:02:09
Commandline: /usr/bin/unattended-upgrade
Remove: libjs-yui3-full:amd64 (3.5.1-1ubuntu3), python3-pexpect:amd64 (4.0.1-1), python3-distro-info:amd64 (0.14build1), libwebp5:amd64 (0.4.4-1), python3-roman:amd64 (2.0.0-2), python3-html5lib:amd64 (0.999-4), squid-langpack:amd64 (20150704-1), python3-ecdsa:amd64 (0.13-2), python3-dnspython:amd64 (1.12.0-0ubuntu3), python3-mimeparse:amd64 (0.1.4-1build1), libirs-export141:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.12), fonts-font-awesome:amd64 (4.5.0~dfsg-1), python3-zope.interface:amd64 (4.1.3-1build1), python3-djorm-ext-pgarray:amd64 (1.2-0ubuntu2), javascript-common:amd64 (11), python3-formencode:amd64 (1.3.0-0ubuntu5), dbconfig-pgsql:amd64 (2.0.4ubuntu1), os-prober:amd64 (1.70ubuntu3.3), python3-seamicroclient:amd64 (0.4.0-1ubuntu1), python3-babel:amd64 (1.3+dfsg.1-6), docutils-common:amd64 (0.12+dfsg-1), python3-maas-provisioningserver:amd64 (2.3.0-6434-gd354690-0ubuntu1~16.04.1), libibverbs1:amd64 (1.1.8-1.1ubuntu2), python3-sphinx:amd64 (1.3.6-2ubuntu1.1), libfreeipmi16:amd64 (1.4.11-1.1ubuntu4~0.16.04), libaprutil1:amd64 (1.5.4-1build1), libipmidetect0:amd64 (1.4.11-1.1ubuntu4~0.16.04), python3-netaddr:amd64 (0.7.18-1), grub-common:amd64 (2.02~beta2-36ubuntu3.17), libirs141:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.12), squid-common:amd64 (3.5.12-1ubuntu7.5), authbind:amd64 (2.1.1+nmu1), freeipmi-common:amd64 (1.4.11-1.1ubuntu4~0.16.04), python3-pam:amd64 (0.4.2-13.2ubuntu2), python3-pbr:amd64 (1.8.0-4ubuntu1), python3-pil:amd64 (3.1.2-0ubuntu1.1), python3-simplestreams:amd64 (0.1.0~bzr426-0ubuntu1.2), libaprutil1-dbd-sqlite3:amd64 (1.5.4-1build1), python3-netifaces:amd64 (0.10.4-0.1build2), python3-django-maas:amd64 (2.3.0-6434-gd354690-0ubuntu1~16.04.1), libjs-jquery:amd64 (1.11.3+dfsg-4), libjs-underscore:amd64 (1.7.0~dfsg-1ubuntu1), libjs-yui3-min:amd64 (3.5.1-1ubuntu3), python3-curtin:amd64 (17.1-11-ga4c9636b-0ubuntu1~16.04.1), python-django-common:amd64 (1.8.7-1ubuntu5.8), python3-service-identity:amd64 (16.0.0-2), bind9utils:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.12), librdmacm1:amd64 (1.0.21-1), libconfig-general-perl:amd64 (2.60-1), python3-jsonschema:amd64 (2.5.1-4), maas-common:amd64 (2.3.0-6434-gd354690-0ubuntu1~16.04.1), python3-txtftp:amd64 (0.1~bzr42-0ubuntu2), apache2-data:amd64 (2.4.18-2ubuntu3.10), python3-crochet:amd64 (1.4.0-0ubuntu2), python3-httplib2:amd64 (0.9.1+dfsg-1), python3-iso8601:amd64 (0.1.11-1), pxelinux:amd64 (3:6.03+dfsg-11ubuntu1), python3-sphinx-rtd-theme:amd64 (0.1.9-1), python3-pyparsing:amd64 (2.0.3+dfsg1-1ubuntu0.1), libpaper1:amd64 (1.1.24+nmu4ubuntu1), libjs-yui3-common:amd64 (3.5.1-1ubuntu3), python3-twisted:amd64 (16.0.0-1ubuntu0.2), python3-simplejson:amd64 (3.8.1-1ubuntu2), ntp:amd64 (1:4.2.8p4+dfsg-3ubuntu5.9), libllvm4.0:amd64 (1:4.0-1ubuntu1~16.04.2), libapr1:amd64 (1.5.2-3), maas-rack-controller:amd64 (2.3.0-6434-gd354690-0ubuntu1~16.04.1), libipmiconsole2:amd64 (1.4.11-1.1ubuntu4~0.16.04), libaprutil1-ldap:amd64 (1.5.4-1build1), maas-cli:amd64 (2.3.0-6434-gd354690-0ubuntu1~16.04.1), distro-info:amd64 (0.14build1), maas-dns:amd64 (2.3.0-6434-gd354690-0ubuntu1~16.04.1), libisccfg-export140:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.12), python3-pyvmomi:amd64 (5.5.0-2014.1.1-3), sphinx-rtd-theme-common:amd64 (0.1.9-1), python3-petname:amd64 (2.0-0ubuntu1~16.04), python3-alabaster:amd64 (0.7.7-1), python3-maas-client:amd64 (2.3.0-6434-gd354690-0ubuntu1~16.04.1), tgt:amd64 (1:1.0.63-1ubuntu1.1), python3-docutils:amd64 (0.12+dfsg-1), postgresql:amd64 (9.5+173ubuntu0.1), python3-tempita:amd64 (0.5.2-1build1), python3-bson-ext:amd64 (3.2-1build1), python3-convoy:amd64 (0.2.1+bzr39-1), maas-region-controller:amd64 (2.3.0-6434-gd354690-0ubuntu1~16.04.1), libpaper-utils:amd64 (1.1.24+nmu4ubuntu1), ieee-data:amd64 (20150531.1), maas-dhcp:amd64 (2.3.0-6434-gd354690-0ubuntu1~16.04.1), python3-django-piston3:amd64 (0.3~rc2-3ubuntu1), liblua5.1-0:amd64 (5.1.5-8ubuntu1), python-babel-localedata:amd64 (1.3+dfsg.1-6), python3-bson:amd64 (3.2-1build1), dbconfig-common:amd64 (2.0.4ubuntu1), libwebpmux1:amd64 (0.4.4-1), python3-django:amd64 (1.8.7-1ubuntu5.8), formencode-i18n:amd64 (1.3.0-0ubuntu5), bind9:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.12), maas-region-api:amd64 (2.3.0-6434-gd354690-0ubuntu1~16.04.1), python3-psycopg2:amd64 (2.6.1-1build2), fonts-lato:amd64 (2.0-1), squid:amd64 (3.5.12-1ubuntu7.5), python3-paramiko:amd64 (1.16.0-1ubuntu0.2), python3-tz:amd64 (2014.10~dfsg1-0ubuntu2), python3-ptyprocess:amd64 (0.5-1), curtin-common:amd64 (17.1-11-ga4c9636b-0ubuntu1~16.04.1), libdebian-installer4:amd64 (0.102ubuntu1.1), apache2-bin:amd64 (2.4.18-2ubuntu3.10), maas-proxy:amd64 (2.3.0-6434-gd354690-0ubuntu1~16.04.1), libsgutils2-2:amd64 (1.40-0ubuntu1), sg3-utils:amd64 (1.40-0ubuntu1), python3-oauth:amd64 (1.0.1-5), python3-pygments:amd64 (2.1+dfsg-1), avahi-utils:amd64 (0.6.32~rc+dfsg-1ubuntu2.3), libjs-modernizr:amd64 (2.6.2+ds1-1ubuntu1), sphinx-common:amd64 (1.3.6-2ubuntu1.1), libopts25:amd64 (1:5.18.7-3), syslinux-common:amd64 (3:6.03+dfsg-11ubuntu1), apache2:amd64 (2.4.18-2ubuntu3.10), isc-dhcp-server:amd64 (4.3.3-5ubuntu12.10), python3-lxml:amd64 (3.5.0-1ubuntu0.1), apache2-utils:amd64 (2.4.18-2ubuntu3.10), libecap3:amd64 (1.0.1-3ubuntu3), libjs-sphinxdoc:amd64 (1.3.6-2ubuntu1.1), archdetect-deb:amd64 (1.117ubuntu2.3), freeipmi-tools:amd64 (1.4.11-1.1ubuntu4~0.16.04), python3-pyasn1-modules:amd64 (0.0.7-0.1), libjs-angularjs:amd64 (1.2.28-1ubuntu2), python3-sqlparse:amd64 (0.1.18-1), python3-attr:amd64 (15.2.0-1), python3-bs4:amd64 (4.4.1-1)
End-Date: 2019-04-24 06:05:08

Pastbin as well:
https://pastebin.canonical.com/p/wwfmG53rNC/

[Andres Rodriguez (andreserl)]

Hi Chris,

Please provide more context on this. We specifically need logs on what apt commands were executed, how, and the full process log of the terminal (e.g. we wanna see what apt has done).

That said, we've never seen anything like this before.

TO recover, you should just be able to stop all maas server, apply your db backup, restart maas services and should be good to go.

[James Troup (elmo)]

adding '-o debug::pkgproblemresolver=true' to apt commands may help here; e.g. if you try to reinstall maas.

[Chris Sanders (chris.sanders)]

Attached dpkg.log

[Chris Sanders (chris.sanders)]

attached apt term.log

[Chris Sanders (chris.sanders)]

attached apt history.log

[Chris Sanders (chris.sanders)]

That's all relevant logs I could think of, when I am able to do a reinstall I'll include the above recommended debug option as well. If there's something I didn't think of let me know and I'll provide that as well.

[Chris Sanders (chris.sanders)]

I've reinstalled with: sudo apt install maas -o debug::pkgproblemresolver=true 2>&1 | tee install.log

As the attached log shows there were no errors, clean install.

After following the restore and restarting the services juju returned the following error:
ERROR opening environment: unexpected: ServerError: 409 Conflict (<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  <head>
    <title>Error: Conflict Error</title>
  </head>
  <body>
    <h2>
      Conflict error. Try your request again, as it will most likely succeed.
    </h2>
  </body>
</html>
)

Restarting the machine has restored normal operation.

[Andres Rodriguez (andreserl)]

After having a look at all the provided logs, I see the following:

1. On dpkg.log I see that packages where installed and removed:

 - on 2019-04-19, non-maas related packages were configured (likely upgraded)
 - on 2019-04-24, maas related packages were marked for removal, and removed, e.g:

2019-04-24 06:02:09 startup packages remove
2019-04-24 06:02:09 status installed maas-region-controller:all 2.3.0-6434-gd354690-0ubuntu1~16.04.1
2019-04-24 06:02:11 remove maas-region-controller:all 2.3.0-6434-gd354690-0ubuntu1~16.04.1 <none>

2. On term.log, I see the counterparts of the above, were:

 - on 2019-04-19 packages were upgraded and MAAS wasn't removed
 - on 2019-04-24 I see that MAAS packages were removed, but nothing was upgraded nor updated.

3. On history.log, I confirm what I see the above, which is that:

 - on 2019-04-19 packages were ntfs-3g:amd64 was upgraded, and nothing removed automatically.
 - on 2019-04-24 MAAS related packages were removed.

From looking at the above, nothing really tells me what could be wrong and why MAAS packages were removed. However, the one thing to notice here is that MAAS 2.3.0-6434-gd354690 was the one removed. The interesting part, is the following:

4. From looking at install.log things turn interesting. This is because what's installed is MAAS 2.3.5-6511-gf466fdb-0ubuntu1.

So, from looking at all of the above I can conclude that:

a. There's not enough information why MAAS got removed.
b. The installed MAAS version on the system was *2.3.0-6434-gd354690* while the version available in -updates was *2.3.5-6511-gf466fdb-0ubuntu1*. This to me means that unattended upgrades never really upgraded the MAAS version with the latest in -updates (e.g. it seems this MAAS could've been installed when 2.3.0 was in -updates).
c. More interestingly, is that:
  - In previous unattended upgrades, no MAAS packages were removed, so why in the latest?
  - When MAAS was removed, no other packages were installed, which if there were to have been a conflict that would have caused MAAS to be removed, I would have expected a dependency would have conflicted and installed instead, which would have caused the upgrade? Unless unattended upgrades simply managed to remove while not upgrade a given dep?
 - When MAAS was manually installed, the installation happened just fine without really any issues, so if there would've been dependency issues, I would have expected the install to complain.

So from the above, it would seem that someone or something marked the MAAS package for removal, and doesn't seem related to a dependency because we can see no dep issues, or new deps installed which would've caused this. That said, I'm not certain how unattended upgrades works internally so not sure why it never did upgrade the MAAS version in previous upgrades, and why it removed it in the latest.

[Andres Rodriguez (andreserl)]

Also, the big questions would be:

1. Why was MAAS never previously been upgraded if 2.3.5 (from 2.3.0) has been in the archive forever. Or why this issue was never seen before if the same applies.
2. There are no reported broken packages, or newly installed packages when MAAS was removed, nor removed packages when MAAS was manually installed later on. So would it be possible the package was marked for removal by something else?

[Steve Langasek (vorlon)]

Balint, can you comment here on the unattended-upgrades autoremoval behavior? I think the intended behavior is only to remove packages that have newly become unused as part of an upgrade, is that correct?

[Balint Reczey (rbalint)]

@vorlon: Yes, but there is (was) a bug where u-u removes auto removable but not newly auto removable packages when an upgrade step fails and leaves a dirty cache behind: LP: #1824341 .
The fix is scheduled to be released for Xenial next week.

@chris.sanders: To confirm that this happened please attach the relevant part of:
/var/log/unattended-upgrades/unattended-upgrades.log

[Chris Sanders (chris.sanders)]

requested logging during the month of the error

[Chris Sanders (chris.sanders)]

dpk logging from the same time period

[Chris Sanders (chris.sanders)]

@rbalint I've included the logs but I don't see any Traces as mentioned in LP: #1824341

[Balint Reczey (rbalint)]

@chris.sanders: Since this happended on 2019-04-24 I assume the system still had u-u 0.90ubuntu0.10 installed. In that case it may have hit LP: #1803749.
If this is the case, the latest versions in all releases are fixed now.