DistUpgradeViewNonInteractive crashes / requires interaction

[Paul Larson]

[Impact]
The NonInteractive view of ubuntu-release-upgrader doesn't work so it is hard to automate upgrade testing.

[Test Case]
You need to create a situation where you'll receive a conffile prompt since the handling of those is broken.

1) On a bionic system modify /etc/update-manager/release-upgrades so that Prompt=normal
2) Test an upgrade from bionic to cosmic or disco i.e. run do-release-upgrade --frontend DistUpgradeViewNonInteractive
3) Observe the upgrade hang on the '/etc/update-manager/release-upgrades' conffifle and the following in /var/log/dist-upgrade/main.log
"2019-05-10 21:21:21,313 WARNING got a conffile-prompt from dpkg for file: '/etc/update-manager/release-upgrades'
2019-05-10 21:21:26,319 ERROR error 'a bytes-like object is required, not 'str'' when trying to write to the conffile"

With the version of the release-upgrader from -proposed you'll no longer observe the upgrade process hanging.

[Regression Potential]
The fix is making it so that a byte object is passed to the prompt instead of a string one so there really isn't one, however additional logging which could result in a Traceback if the syntax of that is wrong.

[Original Description]
I'm trying to do some automated testing that involved upgrading a system from xenial to bionic, so I need it to not ask for user input.

Before running do-release-upgrade, the system got a fresh dist-upgrade and reboot.

To avoid interactive responses, I'm using:
$ sudo do-release-upgrade -d -f DistUpgradeViewNonInteractive

Part way through the upgrade, I do get prompted for something though:
Preparing to unpack .../apt_1.6.3ubuntu0.1_amd64.deb ...

Unpacking apt (1.6.3ubuntu0.1) over (1.2.27) ...

Processing triggers for libc-bin (2.27-3ubuntu1) ...

/sbin/ldconfig.real: Warning: ignoring configuration file that cannot be opened: /etc/ld.so.conf.d/x86_64-linux-gnu_EGL.conf: No such file or directory

/sbin/ldconfig.real: Warning: ignoring configuration file that cannot be opened: /etc/ld.so.conf.d/x86_64-linux-gnu_GL.conf: No such file or directory

Setting up apt (1.6.3ubuntu0.1) ...

Installing new version of config file /etc/apt/apt.conf.d/01autoremove ...

Configuration file '/etc/cron.daily/apt-compat'

 ==> Deleted (by you or by a script) since installation.

 ==> Package distributor has shipped an updated version.

   What would you like to do about it ? Your options are:

    Y or I : install the package maintainer's version

    N or O : keep your currently-installed version

      D : show the differences between the versions

      Z : start a shell to examine the situation

 The default action is to keep your current version.

Comparing the sha1 of apt-compat before the upgrade to another xenial system that has been unmodified, they are the same:

In /var/log/dist-upgrade/main.log, I also found this:
2018-10-04 14:20:24,575 WARNING got a conffile-prompt from dpkg for file: '/etc/cron.daily/apt-compat'
2018-10-04 14:20:29,580 ERROR error 'a bytes-like object is required, not 'str'' when trying to write to the conffile

[Brian Murray (brian-murray)]

If I remember correctly David was having a similar issue when upgrading some charm systems. Is that right David?

[David Ames (thedac)]

Yes, for juju series upgrades.

The following will work around the problem:

echo 'DPkg::options { "--force-confdef"; };' > /etc/apt/apt.conf.d/50unattended-upgrades

I'll leave the decision on if this remains a bug up to others.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Brian Murray (brian-murray)]

The following error message is from the conffile function in DistUpgradeViewNonInteractive.py

2018-10-04 14:20:29,580 ERROR error 'a bytes-like object is required, not 'str'' when trying to write to the conffile

[Brian Murray (brian-murray)]

I tested an upgrade from xenial to bionic today and did not receive a conffile prompt regarding /etc/cron.daily/apt-compat and part of the output you've pasted leads me to believe the file did not exist on the system which you were upgrading. Have you run into this any more?

Additionally, I've sorted out the issue in ubuntu-release-upgrader for Eoan. If you want it SRU'ed let us know.

[Brian Murray (brian-murray)]

Well I'm fairly certain this issue, DistUpgradeViewNonInteractive not being non-interactive, is contributing to automatic upgrade testing so I'll be SRU'ing this change everywhere.

[Brian Murray (brian-murray)]

To elaborate more I seen the following in the console output of a job which upgrades from bionic to cosmic:

Setting up ubuntu-release-upgrader-core (1:18.10.11.6) ...

Configuration file '/etc/update-manager/release-upgrades'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ? Your options are:
    Y or I : install the package maintainer's version
    N or O : keep your currently-installed version
      D : show the differences between the versions
      Z : start a shell to examine the situation
 The default action is to keep your current version.
*** release-upgrades (Y/I/N/O/D/Z) [default=N] ? dpkg: error: dpkg status database is locked by another process
auto-upgrade [06:09:35]: ERROR: ERROR: Something went wrong with the upgrade.

release-upgrades has had Prompt changed from lts to normal so that we can test the upgrade to cosmic and then we get the bytes like object crash.

[Paul Larson (pwlars)]

For the xenial->bionic question, I had already implemented the workaround mentioned here to use Dpkg::Options::="--force-confdef" across all upgrade tests, which has been working fine. I can remove the workaround and try it if you are specifically interested in finding out whether it's necessary on the upgrade to bionic though.

[Brian Murray (brian-murray)]

On Fri, May 10, 2019 at 08:13:47PM -0000, Paul Larson wrote:
> For the xenial->bionic question, I had already implemented the
> workaround mentioned here to use Dpkg::Options::="--force-confdef"
> across all upgrade tests, which has been working fine. I can remove the
> workaround and try it if you are specifically interested in finding out
> whether it's necessary on the upgrade to bionic though.

Yes, the foundations team is interested. If there is a package creating
a conffile prompt during a release upgrade we'd like to know what
package it is and why.

--
Brian Murray

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:19.10.5

---------------
ubuntu-release-upgrader (1:19.10.5) eoan; urgency=medium

  * DistUpgrade/DistUpgradeViewNonInteractive.py: when responding to a
    conffile prompt with "n" use bytes not a string. (LP: #1796193)
  * DistUpgrade/DistUpgradeController.py: set a default value for devRelease
    even if there are no options specified. (LP: #1827441)

 -- Brian Murray <email address hidden> Fri, 10 May 2019 13:30:07 -0700

[Łukasz Zemczak (sil2100)]

Hello Paul, or anyone else affected,

Accepted ubuntu-release-upgrader into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:19.04.16.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Paul, or anyone else affected,

Accepted ubuntu-release-upgrader into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:18.10.11.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Verified for an upgrade from Bionic to Cosmic:

bdmurray@clean-bionic-amd64:~$ grep conffile-prompt /var/log/dist-upgrade/main.log
2019-05-14 16:19:53,226 WARNING got a conffile-prompt from dpkg for file: '/etc/update-manager/release-upgrades'
2019-05-14 16:19:58,232 WARNING replied no to the conffile-prompt for file: '/etc/update-manager/release-upgrades'
bdmurray@clean-bionic-amd64:~$ head -n10 /var/log/dist-upgrade/main.log
2019-05-14 16:03:25,448 INFO Using config files '['./DistUpgrade.cfg']'
2019-05-14 16:03:25,448 INFO uname information: 'Linux clean-bionic-amd64 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64'
2019-05-14 16:03:25,783 INFO apt version: '1.6.10'
2019-05-14 16:03:25,784 INFO python version: '3.6.7 (default, Oct 22 2018, 11:32:17)
[GCC 8.2.0]'
2019-05-14 16:03:25,786 INFO release-upgrader version '18.10.11.8' started
2019-05-14 16:03:25,799 INFO locale: 'en_US' 'UTF-8'
2019-05-14 16:03:25,871 DEBUG setting up environ for non-interactive use
2019-05-14 16:03:26,083 DEBUG Using 'DistUpgradeViewNonInteractive' view
2019-05-14 16:03:26,117 DEBUG enable dpkg --force-overwrite

[Brian Murray (brian-murray)]

I also verified this for an upgrade from Bionic to Disco, after hacking at meta-release-proposed.

bdmurray@clean-bionic-amd64:~$ grep conffile /var/log/dist-upgrade/main.log
2019-05-14 22:37:36,972 WARNING got a conffile-prompt from dpkg for file: '/etc/update-manager/release-upgrades'
2019-05-14 22:37:41,977 WARNING replied no to the conffile-prompt for file: '/etc/update-manager/release-upgrades'
bdmurray@clean-bionic-amd64:~$ head -n10 /var/log/dist-upgrade/main.log
2019-05-14 22:18:11,371 INFO Using config files '['./DistUpgrade.cfg.bionic']'
2019-05-14 22:18:11,371 INFO uname information: 'Linux clean-bionic-amd64 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64'
2019-05-14 22:18:11,702 INFO apt version: '1.6.10'
2019-05-14 22:18:11,702 INFO python version: '3.6.7 (default, Oct 22 2018, 11:32:17)
[GCC 8.2.0]'
2019-05-14 22:18:11,704 INFO release-upgrader version '19.04.16.5' started
2019-05-14 22:18:11,715 INFO locale: 'en_US' 'UTF-8'
2019-05-14 22:18:11,783 DEBUG setting up environ for non-interactive use
2019-05-14 22:18:11,842 DEBUG Using 'DistUpgradeViewNonInteractive' view
2019-05-14 22:18:11,877 DEBUG enable dpkg --force-overwrite

[Paul Larson (pwlars)]

From a xenial system, I tried upgrading to bionic with do-release-upgrade with -f DistUpgradeViewNonInteractive and *not* setting up the force-confdef option, and it asks for input

Here's where it got stuck waiting for input:

Configuration file '/etc/ntp.conf'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ? Your options are:
    Y or I : install the package maintainer's version
    N or O : keep your currently-installed version
      D : show the differences between the versions
      Z : start a shell to examine the situation
 The default action is to keep your current version.
*** ntp.conf (Y/I/N/O/D/Z) [default=N] ?

[Brian Murray (brian-murray)]

On Thu, May 16, 2019 at 03:44:18PM -0000, Paul Larson wrote:
> >From a xenial system, I tried upgrading to bionic with do-release-
> upgrade with -f DistUpgradeViewNonInteractive and *not* setting up the
> force-confdef option, and it asks for input

The fix for this in Bionic is still In Progress, it'll hopefully make it
to -proposed this week. When it becomes available in -proposed you'll
want to use do-release-upgrade -p -f DistUpgradeViewNonInteractive.

--
Brian Murray

[Łukasz Zemczak (sil2100)]

Hello Paul, or anyone else affected,

Accepted ubuntu-release-upgrader into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:18.04.33 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Paul Larson (pwlars)]

Strange, that link seems to suggest it is in proposed already, but when I try to run 'do-release-upgrade -p -f DistUpgradeViewNonInteractive', I don't seem to get that. Do I need to do some other manual step to get it first, or do I need it from xenial instead since that's where I'm coming from? Even though I used -p, proposed doesn't seem to be on in /etc/apt/source.list
(after I got the error from above again, I killed the upgrader and checked this)
$ apt-cache policy ubuntu-release-upgrader-core
ubuntu-release-upgrader-core:
  Installed: 1:18.04.32
  Candidate: 1:18.04.32
  Version table:
 *** 1:18.04.32 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages
        100 /var/lib/dpkg/status
     1:18.04.17 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu bionic/main i386 Packages

[Brian Murray (brian-murray)]

The command do-release-upgrade downloads a tarball of ubuntu-release-upgrader from the release to which you are upgrading e.g. http://archive.ubuntu.com/ubuntu/dists/bionic-proposed/main/dist-upgrader-all/current/bionic.tar.gz. That tarball is then extracted into /tmp/ubuntu-release-upgrader-$TEMP/. To confirm you are getting the right version of the tarball you can check DistUpgradeVersion.py in that folder. It should contain 18.04.33. It looks right to me:

bdmurray@clean-xenial-amd64:~$ cd /tmp/ubuntu-release-upgrader-pn4lfyfj/
bdmurray@clean-xenial-amd64:/tmp/ubuntu-release-upgrader-pn4lfyfj$ cat DistUpgradeVersion.py
VERSION = '18.04.33'

[Paul Larson (pwlars)]

Ok, yeah I see it running:
/usr/bin/python3 /tmp/ubuntu-release-upgrader-n98wh3a9/bionic --mode=server --frontend=DistUpgradeViewNonInteractive
and
$ sudo cat /tmp/ubuntu-release-upgrader-n98wh3a9/DistUpgradeVersion.py
VERSION = '18.04.33'

after letting it run a while, I did see it seemed to be stuck on ntp and asking me what I wanted to do. I Switched windows but I was pretty sure I didn't press anything else. It had progressed when I went back, but I wouldn't have expected it to be asking at all. I'll try to reproduce again, but it takes a while to get there.

[Brian Murray (brian-murray)]

On Fri, May 17, 2019 at 07:44:50PM -0000, Paul Larson wrote:
> Ok, yeah I see it running:
> /usr/bin/python3 /tmp/ubuntu-release-upgrader-n98wh3a9/bionic --mode=server --frontend=DistUpgradeViewNonInteractive
> and
> $ sudo cat /tmp/ubuntu-release-upgrader-n98wh3a9/DistUpgradeVersion.py
> VERSION = '18.04.33'
>
> after letting it run a while, I did see it seemed to be stuck on ntp and
> asking me what I wanted to do. I Switched windows but I was pretty sure
> I didn't press anything else. It had progressed when I went back, but I
> wouldn't have expected it to be asking at all. I'll try to reproduce
> again, but it takes a while to get there.

There's a 5 second sleep or so after the dialog appears before a
response is given and the comment indicates that is due to a race
condition. I wasn't brave enough to mess with that. ;-)

--
Brian Murray

[Paul Larson (pwlars)]

hah, ok that makes more sense then. Looks like this works, thanks!

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:19.04.16.5

---------------
ubuntu-release-upgrader (1:19.04.16.5) disco; urgency=medium

  [ Gunnar Hjalmarsson ]
  * DistUpgrade/window_main.ui: Really update for 19.04 (LP: #1825733).

  [ Brian Murray ]
  * DistUpgrade/DistUpgradeViewNonInteractive.py: when responding to a
    conffile prompt with "n" use bytes not a string. (LP: #1796193)

 -- Brian Murray <email address hidden> Fri, 10 May 2019 15:19:00 -0700

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:18.10.11.8

---------------
ubuntu-release-upgrader (1:18.10.11.8) cosmic; urgency=medium

  * DistUpgrade/DistUpgradeViewNonInteractive.py: when responding to a
    conffile prompt with "n" use bytes not a string. (LP: #1796193)

 -- Brian Murray <email address hidden> Fri, 10 May 2019 16:14:50 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:18.04.33

---------------
ubuntu-release-upgrader (1:18.04.33) bionic; urgency=medium

  * DistUpgrade/DistUpgradeViewNonInteractive.py: when responding to a
    conffile prompt with "n" use bytes not a string. (LP: #1796193)
  * debian/control: add an explicit python3-distro-info dependency to
    python3-distupgrade as it is now required by DistUpgradeController.
    (LP: #1824866)
  * pre-build.sh: stop updating demotions for artful since it is no longer
    available at archive.ubuntu.com and if we do update then demotions becomes
    an empty file which is wrong.

 -- Brian Murray <email address hidden> Wed, 15 May 2019 10:37:15 -0700

[Radoslav Stefanov (radoslav-r)]

This is still causing problems when upgrading from Xenial to Bionic. Even if I use do-release-upgrade -p -f DistUpgradeViewNonInteractive

It gets stuck asking for a prompt at:

Configuration file '/etc/ntp.conf'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ? Your options are:
    Y or I : install the package maintainer's version
    N or O : keep your currently-installed version
      D : show the differences between the versions
      Z : start a shell to examine the situation
 The default action is to keep your current version.
*** ntp.conf (Y/I/N/O/D/Z) [default=N] ? n

[Andrey Arapov (andrey-arapov)]

Each time I am upgrading Xenial to Bionic I'm getting exactly the same message after few minutes:

"*** ntp.conf (Y/I/N/O/D/Z) [default=N] ? "

Can this fix get delivered to Ubuntu Xenial?

ubuntu-release-upgrader-core 1:16.04.26

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.