packages with only upstart+systemd without sysvinit fail to upgrade with do-release-upgrade: upgrade tarballs ship obsolete invoke-rc.d

[Pavel-II]

After upgrade to 15.10

SRU TEST CASE:
 - Take a vivid schroot, cloud image, container or similar
 - Install ofono, lxc, or nfs-common
 - Run "do-release-upgrade -d"
 - With current tarball this fails with something like
   initctl: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused
   invoke-rc.d: unknown initscript, /etc/init.d/lxc not found.
 - With fixed upgrade tarball the upgrade should succeed.

ProblemType: Package
DistroRelease: Ubuntu 15.10
Package: nfs-common 1:1.2.8-9ubuntu10
ProcVersionSignature: Ubuntu 4.2.0-16.19-generic 4.2.3
Uname: Linux 4.2.0-16-generic x86_64
ApportVersion: 2.19.1-0ubuntu2
Architecture: amd64
Date: Sun Oct 11 04:41:20 2015
ErrorMessage: подпроцесс установлен сценарий post-installation возвратил код ошибки 100
InstallationDate: Installed on 2009-11-30 (2140 days ago)
InstallationMedia: Ubuntu 9.10 "Karmic Koala" - Release amd64 (20091027.1)
RelatedPackageVersions:
 dpkg 1.18.2ubuntu4
 apt 1.0.10.2ubuntu1
SourcePackage: nfs-utils
Title: package nfs-common 1:1.2.8-9ubuntu10 failed to install/upgrade: подпроцесс установлен сценарий post-installation возвратил код ошибки 100
UpgradeStatus: Upgraded to wily on 2015-10-11 (0 days ago)

[Ransom stark (neil-stark-developments)]

Thank you for taking the time to report this issue and helping to make Ubuntu better. We noticed that some of the sentences in this bug report are not in English. If they were translated to English they would be more understandable to triagers. Could you please translate them?

[Pavel-II (dzy4)]

ErrorMessage: подпроцесс установлен сценарий post-installation возвратил код ошибки 100
and
подпроцесс установлен сценарий post-installation возвратил код ошибки 100

The bottom line is that the installation process (subprocess post-installation) was completed with an error code 100

[Steve Langasek (vorlon)]

The full error in the log is:

Настраивается пакет nfs-common (1:1.2.8-9ubuntu10) …
initctl: Не удается подключиться к Upstart: Failed to connect to socket /com/ubuntu/upstart: В соединении отказано
invoke-rc.d: unknown initscript, /etc/init.d/gssd not found.
dpkg: ошибка при обработке пакета nfs-common (--configure):
 подпроцесс установлен сценарий post-installation возвратил код ошибки 100

This is supposed to have been fixed in the upload of nfs-utils 1:1.2.8-9ubuntu10, which is the exact version you're installing; invoke-rc.d is not supposed to fail in this case.

Martin, can you take a look at this and see if we need to be adding a versioned dependency on sysv-rc for such cases?

[Martin Pitt (pitti)]

A versioned dep can't explain this fully, as in the upgrade log wily's sysv-rc gets installed and configured much earlier than nfs-common.

I tried to reproduce this in a VM, but no luck so far. I tried with (touch /etc/krb5.keytab) and without (default) running gssd.

sudo apt install -y rpcbind nfs-common
sudo touch /etc/krb5.keytab
sudo systemctl start rpc-gssd # or reboot
echo 'deb http://archive.ubuntu.com/ubuntu wily main restricted universe' | sudo tee /etc/apt/sources.list
sudo apt update
# same order as in reported log
sudo apt install -y sysv-rc initscripts systemd systemd-sysv
sudo apt install -y rpcbind nfs-common
# alternatively:
sudo apt full-upgrade

I also tried with installing/upgrading upstart and upstart-bin, no change.

I never see these messages:

insserv: warning: current start runlevel(s) (2 3 4 5 S) of script `rpcbind' overrides LSB defaults (S).
initctl: Не удается подключиться к Upstart: Failed to connect to socket /com/ubuntu/upstart: В соединении отказано
invoke-rc.d: unknown initscript, /etc/init.d/gssd not found.

so something in your install is different from my fresh vivid install.

@rich: Can you please tell me more about the history of this system? I. e. which version did you originally install (looks like Ubuntu 9.10?), which upgrades did you do, etc.?

[Martin Pitt (pitti)]

Tried again from trusty:

sudo apt install -y rpcbind nfs-common
sudo sed -i '/NEED_GSSD/ s/=/=yes/' /etc/default/nfs-common
sudo initctl start gssd
echo 'deb http://archive.ubuntu.com/ubuntu vivid main restricted universe' | sudo tee /etc/apt/sources.list
sudo apt update
sudo apt -y full-upgrade
sudo touch /etc/krb5.keytab # to enable gssd
sudo reboot
echo 'deb http://archive.ubuntu.com/ubuntu wily main restricted universe' | sudo tee /etc/apt/sources.list
sudo apt update
sudo apt -y full-upgrade

This still works fine, and I still don't get all these error messages.

@rich: Can you please copy&paste the output of these commands:

   md5sum `which invoke-rc.d`
   sudo sh -x /usr/sbin/invoke-rc.d gssd restart
   sudo apt-get install --reinstall nfs-common

Thanks!

[Pavel-II (dzy4)]

Full log translation:

Configures the package nfs-common (1:1.2.8-9ubuntu10) …
initctl: Cannot connect to Upstart: Failed to connect to socket /com/ubuntu/upstart:
Connection refused invoke-rc.d: unknown initscript, /etc/init.d/gssd not found.
dpkg: an error processing package nfs-common (--configure):
subprocess installed scenario post-installation return error code 100

@Martin Pitt (pitti):
md5sum `which invoke-rc.d` : d6ecd6a53a96b6de44ab21a4efa6d8e5

sudo sh -x /usr/sbin/invoke-rc.d gssd restart && sudo apt-get install --reinstall nfs-common:
+ RUNLEVELHELPER=/sbin/runlevel
+ POLICYHELPER=/usr/sbin/policy-rc.d
+ INITDPREFIX=/etc/init.d/
+ RCDPREFIX=/etc/rc
+ BEQUIET=
+ MODE=
+ ACTION=
+ FALLBACK=
+ NOFALLBACK=
+ FORCE=
+ RETRY=
+ RETURNFAILURE=
+ RC=
+ is_upstart=
+ is_systemd=
+ set +e
+ test 2 -eq 0
+ state=I
+ test 2 -gt 0
+ test I != III
+ verifyparameter gssd
+ test 1 -eq 0
+ test 1 -ne 1
+ return
+ INITSCRIPTID=gssd
+ state=II
+ shift
+ test 1 -gt 0
+ test II != III
+ verifyparameter restart
+ test 1 -eq 0
+ test 1 -ne 1
+ return
+ ACTION=restart
+ state=III
+ shift
+ test 0 -gt 0
+ test III != III
+ unset UPSTART_SESSION
+ which initctl
+ grep -q upstart
+ initctl version
+ test -d /run/systemd/system
+ is_systemd=1
+ UNIT=gssd.service
+ sed s/.*\ //
+ /sbin/runlevel
+ RL=5
+ test ! 0
+ test x5 = x0
+ test x5 = x6
+ test x5 = x0
+ test x5 = x6
+ test x5 != x
+ xargs
+ ls -d -Q /etc/rc5.d/S[0-9][0-9]gssd
+ SLINK=
+ xargs
+ ls -d -Q /etc/rc5.d/K[0-9][0-9]gssd
+ KLINK=
+ xargs
+ ls -d -Q /etc/rcS.d/S[0-9][0-9]gssd
+ SSLINK=
+ verifyrclink
+ doexit=
+ test 0 -gt 0
+ test x != x
+ return 0
+ RC=
+ testexec
+ test 0 -gt 0
+ return 1
+ testexec
+ test 0 -gt 0
+ return 1
+ testexec
+ test 0 -gt 0
+ return 1
+ _executable=
+ [ -n ]
+ [ -n 1 ]
+ systemctl -p LoadState show gssd.service
+ _state=LoadState=loaded
+ [ LoadState=loaded != LoadState=masked ]
+ _executable=1
+ [ 1 = 1 ]
+ test x = x
+ test x = xquery
+ querypolicy
+ policyaction=restart
+ test x = x101
+ test x/usr/sbin/policy-rc.d != x
+ test -x /usr/sbin/policy-rc.d
+ test x = x
+ RC=104
+ return
+ test x = xquery
+ [ -n ]
+ [ 0 != 0 ]
+ clean_ldpreload libeatmydata.so
+ local name=libeatmydata.so oifs=
 found=0 tok= nval= delim=
+ [ = ]
+ IFS=
+ IFS=

+ [ 0 = 1 ]
+ return 0
+ test x != x
+ test 104 -eq 104
+ [ -n ]
+ [ -n 1 ]
+ RC=102
+ setechoactions restart
+ test 1 -gt 1
+ echoaction=
+ test ! -z restart
+ getnextaction restart
+ saction=restart
+ shift
+ ACTION=
+ test ! -z
+ [ -n ]
+ [ -n 1 ]
+ [ -n ]
+ systemctl is-system-running
+ OUT=degraded
+ [ degraded != degraded ]
+ systemctl --quiet is-enabled gssd.service
+ ERR=
+ systemctl restart gssd.service
+ exit 0
Reading package lists… Done
Building dependency tree
Reading state information… Done
update 0, install 0 new packages, reinstall 1 reinstall, mark for delete 0 packages, and 11 packages not upgrade.
Need to upload 0 B/185 kB archives.
After this operation, the volume of the occupied disk space will be 0 B.
(Read database … at the moment installes 591005 files and directories.)
Prepare to unpack …/nfs-common_1%3a1.2.8-9ubuntu10_amd64.deb …
Unpack nfs-common (1:1.2.8-9ubuntu10) to replace (1:1....

[Martin Pitt (pitti)]

OK, thanks. That looks like it's supposed to. So I don't yet know how to reproduce this.

I looked at /usr/sbin/invoke-rc.d again, and the only case where initctl is being called that is not protected by [ -n "$is_upstart" ] is the "are we running upstart?" test at the beginning, but its error messages go to /dev/null. But if you were actually running upstart in 15.04, then (1) the upgrade log would have looked different (you already had systemd-sysv installed, and dmesg shows no init= argument), and communication with /com/ubuntu/upstart should have worked.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Brian Murray (brian-murray)]

There is an errors bucket corresponding to this issue here:

https://errors.ubuntu.com/problem/7f52b91a3a4b716e9d8f59ad80d8681e8479870a

[Vincent Ladeuil (vila)]

/var/log/dist-upgrade/apt-clone_system_state.tar.gz

[Steve Langasek (vorlon)]

there are multiple reports of this issue. marking as high-priority.

[Martin Pitt (pitti)]

bug 1502536 for ofono and bug 1490110 for lxc are essentially the same -- this affects packages which have upstart jobs and systemd units, but not sysv init script. Duplicating and generalizing.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Martin Pitt (pitti)]

Ah, I can finally reproduce this! It seems the crucial difference is using do-release-upgrade; with apt-get or apt it always works fine.

I start from a slightly minimized vivid cloud image.

sudo apt-get install -y ubuntu-release-upgrader-core ofono
sudo do-release-upgrade -d

(using "lxc juju-local" instead of "ofono" as in bug 1490110 works too).

This even works in a schroot with policy-rc.d (which is much faster as it doesn't have to dist-upgrade the kernel etc). The upgrade doesn't fail, but one still sees

Setting up ofono (1.17.bzr6904+15.10.20150928.1-0ubuntu1) ...
Installing new version of config file /etc/dbus-1/system.d/ofono.conf ...
initctl: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused
invoke-rc.d: unknown initscript, /etc/init.d/ofono not found.
invoke-rc.d: policy-rc.d denied execution of start.

[Martin Pitt (pitti)]

This is actually a bug in ubuntu-release-upgrader. The downloaded wily.tar.gz contains *its own* version of invoke-rc.d, and it's wildly out of date (pre-vivid at least). I extracted the tarball and diffed its imported/invoke-rc.d against /usr/sbin/invoke-rc.d and get http://paste.ubuntu.com/12888661/ -- IOW, all the systemd support is completely missing.

[Martin Pitt (pitti)]

I'm afraid I don't know the first thing about how these upgrade tarballs are constructed and how we can update them to the current sysvinit. Michael, Brian, any idea?

[Martin Pitt (pitti)]

Fixed the copy for release-upgrader tarballs: http://bazaar.launchpad.net/~ubuntu-core-dev/ubuntu-release-upgrader/trunk/revision/2922

[Martin Pitt (pitti)]

For the record: to test this while it's in proposed, change /etc/update-manager/meta-release to point to a local copy of http://changelogs.ubuntu.com/meta-release-development with the target changed to -proposed.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1504897

[Brian Murray (brian-murray)]

It's also possible to test this by downloading the proposed version of the release upgrader tarball, extracting it and then running the wily script to run the upgrade process. The proposed version will be available here:

http://archive.ubuntu.com/ubuntu/dists/wily-proposed/main/dist-upgrader-all/current/

[Brian Murray (brian-murray)]

I've tested an upgrade from vivid to wily using the release upgrader from -proposed as I mentioned and see the following in my /var/log/dist-upgrade/main.log file:

Setting up lxc (1.1.4-0ubuntu1) ...^M
Installing new version of config file /etc/apparmor.d/lxc/lxc-default-with-nesting ...^M
Installing new version of config file /etc/bash_completion.d/lxc ...^M
Installing new version of config file /etc/init/lxc.conf ...^M
Setting up lxc dnsmasq configuration.^M
Setting up lxc-templates (1.1.4-0ubuntu1) ...^M

So this looks good to me, thanks!

[Martin Pitt (pitti)]

I also did a test dist-upgrade with the proposed upgrade tarball, with ofono and lxc, and it worked just fine now. Marking verified.

[Iain Lane (laney)]

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:15.10.14

---------------
ubuntu-release-upgrader (1:15.10.14) wily; urgency=medium

  [ Brian Murray ]
  * data/DistUpgrade.cfg: Update kernel version and types that will be
    considered for removal. (LP: #1508539)

  [ Martin Pitt ]
  * DistUpgrade/imported/invoke-rc.d: Import current version with systemd
    support, to fix upgrade failures of packages which only ship upstart and
    systemd units, but no SysV init script (like lxc, ofono, or nfs-common).
    (LP: #1504897)

 -- Martin Pitt <email address hidden> Wed, 21 Oct 2015 23:40:22 +0200

[Martin Pitt (pitti)]

http://changelogs.ubuntu.com/meta-release-development got updated now, and I did a full end-to-end upgrade test, which works fine now.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:16.04.1

---------------
ubuntu-release-upgrader (1:16.04.1) xenial; urgency=medium

  [ Barry Warsaw ]
  * Don't try to import janitor and NvidiaDetector in a package relative
    to this one, to avoid the possibility of dangling symlinks when the
    packages actually containing these modules is not installed.
    (LP: #1485093)
  * debian/python3-distupgrade.links: Removed.

  [ Brian Murray ]
  * DistUpgrade/{EOL,}ReleaseAnnouncement,
    DistUpgrade/DistUpgrade/window_main.ui,
    data/gtkbuilder/DistUpgrade.ui,
    data/DistUpgrade.cfg,
    data/DistUpgrade.cfg.trusty,
    pre-build.sh:
    - updated for 15.10->16.04
  * DistUpgrade/DistUpgradeQuirks.py: add in a quirk to ensure that a linux
    metapackage is installed after the release upgrade. (LP: #1509305)

  [ Michael Vogt ]
  * DistUpgrade/DistUpgradeQuirks.py: improve linux metapackage check and add
    a test for it.

 -- Brian Murray <email address hidden> Tue, 27 Oct 2015 13:45:19 -0700