UIFE - remove ubiquity upgrade option

Bug #1756862 reported by Jean-Baptiste Lallement on 2018-03-19
28
This bug affects 2 people
Affects Status Importance Assigned to Milestone
apt-clone (Ubuntu)
High
Unassigned
Bionic
High
Unassigned
ubiquity (Ubuntu)
High
Brian Murray
Bionic
High
Brian Murray

Bug Description

[Rationale]

The ubiquity upgrade option, which allows one to upgrade an old release of Ubuntu using ubiquity, is crashing when attempting to restore the installed packages. Digging into the matter we discovered that there are a multitude of reports where the upgrade crashes due to a failure to calculate the upgrade and this leaves users in a state from which it is hard to recover. Additionally, there have been few crashes reported about this process which leads us to believe it is not used frequently.

Original Description
--------------------
Test Case:
1. Install 17.10 and apply all updates
2. boot the ISO of Ubuntu Desktop 18.04
3. Launch the install and in the partitioning step, select Upgrade from 17.10 to 18.04
4. Proceed with the rest of the installation

Expected result
The system upgrades successfully to 18.04

Actual result
This crash

ProblemType: CrashDistroRelease: Ubuntu 18.04
Package: apt-clone 0.4.1ubuntu2
ProcVersionSignature: Ubuntu 4.15.0-12.13-generic 4.15.7
Uname: Linux 4.15.0-12-generic x86_64
ApportVersion: 2.20.8-0ubuntu10
Architecture: amd64
CasperVersion: 1.389
Date: Mon Mar 19 12:59:14 2018
ExecutablePath: /usr/bin/apt-clone
InterpreterPath: /usr/bin/python3.6
LiveMediaBuild: Ubuntu 18.04 LTS "Bionic Beaver" - Alpha amd64 (20180317)
PackageArchitecture: all
ProcCmdline: /usr/bin/python3 /usr/bin/apt-clone restore-new-distro /target/ubiquity-apt-clone/apt-clone-state-hostname.tar.gz bionic --destination /target
Python3Details: /usr/bin/python3.6, Python 3.6.5rc1, python3-minimal, 3.6.4-1
PythonArgs: ['/usr/bin/apt-clone', 'restore-new-distro', '/target/ubiquity-apt-clone/apt-clone-state-hostname.tar.gz', 'bionic', '--destination', '/target']
PythonDetails: /usr/bin/python2.7, Python 2.7.14+, python-minimal, 2.7.14-4SourcePackage: apt-clone
Title: apt-clone crashed with SystemError in commit(): installArchives() failed
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

Related branches

Jean-Baptiste Lallement (jibel) wrote :
tags: removed: need-duplicate-check
Changed in apt-clone (Ubuntu):
importance: Undecided → Medium

Mar 19 11:59:14 ubuntu ubiquity: Traceback (most recent call last):
Mar 19 11:59:14 ubuntu ubiquity: File "/usr/bin/apt-clone", line 148, in <module>
Mar 19 11:59:14 ubuntu ubiquity: args.source, args.destination, new_distro=args.new_distro_codename, protect_installed=protect_installed)
Mar 19 11:59:14 ubuntu ubiquity: File "/usr/lib/python3/dist-packages/apt_clone.py", line 501, in restore_state
Mar 19 11:59:14 ubuntu ubiquity: self._restore_package_selection(statefile, targetdir, protect_installed, exclude_pkgs)
Mar 19 11:59:14 ubuntu ubiquity: File "/usr/lib/python3/dist-packages/apt_clone.py", line 658, in _restore_package_selection
Mar 19 11:59:14 ubuntu ubiquity: cache.commit(self.fetch_progress, self.install_progress)
Mar 19 11:59:14 ubuntu ubiquity: File "/usr/lib/python3/dist-packages/apt/cache.py", line 529, in commit
Mar 19 11:59:14 ubuntu ubiquity: raise SystemError("installArchives() failed")
Mar 19 11:59:14 ubuntu ubiquity: SystemError: installArchives() failed
Mar 19 11:59:14 ubuntu /plugininstall.py: log-output -t ubiquity chroot /target umount /sys
Mar 19 11:59:14 ubuntu /plugininstall.py: log-output -t ubiquity chroot /target umount /proc
Mar 19 11:59:14 ubuntu /plugininstall.py: log-output -t ubiquity umount /target/run
Mar 19 11:59:14 ubuntu /plugininstall.py: log-output -t ubiquity umount /target/dev
Mar 19 11:59:14 ubuntu /plugininstall.py: log-output -t ubiquity umount -f /target/proc
Mar 19 11:59:14 ubuntu /plugininstall.py: log-output -t ubiquity umount -f /target/sys
Mar 19 11:59:14 ubuntu /plugininstall.py: log-output -t ubiquity umount -f /target/dev
Mar 19 11:59:14 ubuntu /plugininstall.py: log-output -t ubiquity umount -f /target/run
Mar 19 11:59:14 ubuntu /plugininstall.py: Could not restore packages from the previous install:
Mar 19 11:59:14 ubuntu /plugininstall.py: Traceback (most recent call last):
Mar 19 11:59:14 ubuntu /plugininstall.py: File "/usr/share/ubiquity/plugininstall.py", line 246, in run
Mar 19 11:59:14 ubuntu /plugininstall.py: self.apt_clone_restore()
Mar 19 11:59:14 ubuntu /plugininstall.py: File "/usr/share/ubiquity/plugininstall.py", line 1486, in apt_clone_restore
Mar 19 11:59:14 ubuntu /plugininstall.py: restore_cmd, preexec_fn=install_misc.debconf_disconnect)
Mar 19 11:59:14 ubuntu /plugininstall.py: File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
Mar 19 11:59:14 ubuntu /plugininstall.py: raise CalledProcessError(retcode, cmd)
Mar 19 11:59:14 ubuntu /plugininstall.py: subprocess.CalledProcessError: Command '['apt-clone', 'restore-new-distro', '/target/ubiquity-apt-clone/apt-clone-state-ubuntu.tar.gz', 'bionic', '--destination', '/target']' returned non-zero exit status 1.

information type: Private → Public
description: updated
tags: added: rls-bb-incoming
summary: - apt-clone crashed with SystemError in commit(): installArchives() failed
+ upgrade from 17.10 to 18.04 - apt-clone crashed with SystemError in
+ commit(): installArchives() failed
Changed in apt-clone (Ubuntu):
importance: Medium → High
Jean-Baptiste Lallement (jibel) wrote :

and earlier in syslog

Mar 19 11:59:06 ubuntu ubiquity: Chrooting into
Mar 19 11:59:06 ubuntu ubiquity: /target/
Mar 19 11:59:06 ubuntu ubiquity:
Mar 19 11:59:06 ubuntu ubiquity: dpkg: error: cannot stat pathname '/tmp/apt-dpkg-install-Kthb4J': No such file or directory

tags: removed: rls-bb-incoming
tags: added: id-5ab56b8087bcbfd97826a13b
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apt-clone (Ubuntu):
status: New → Confirmed
Brian Murray (brian-murray) wrote :

From the duplicate report:

It can also be reproduce with the following command
$ sudo apt-clone restore-new-distro \
    /target/ubiquity-apt-clone/apt-clone-state-ubuntu.tar.gz \
    bionic --destination /target

Ubuntu QA Website (ubuntuqa) wrote :

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1756862

tags: added: iso-testing
Brian Murray (brian-murray) wrote :

The traceback from apt-clone can be seen here:

Traceback (most recent call last):
  File "/usr/bin/apt-clone", line 148, in <module>
    args.source, args.destination, new_distro=args.new_distro_codename, protect_installed=protect_installed)
  File "/usr/lib/python3/dist-packages/apt_clone.py", line 501, in restore_state
    self._restore_package_selection(statefile, targetdir, protect_installed, exclude_pkgs)
  File "/usr/lib/python3/dist-packages/apt_clone.py", line 656, in _restore_package_selection
    self._restore_package_selection_in_cache(statefile, cache, protect_installed, exclude_pkgs)
  File "/usr/lib/python3/dist-packages/apt_clone.py", line 633, in _restore_package_selection_in_cache
    resolver.resolve()
apt_pkg.Error: E:Unable to correct problems, you have held broken packages.

Brian Murray (brian-murray) wrote :

Running it with --simulate we can see something weird is going on:

sudo APT_CLONE_DEBUG_RESOLVER=1 apt-clone restore-new-distro /tmp/apt-clone-state-ubuntu.tar.gz bionic --destination / --simulate
missing: adwaita-icon-theme-full,fonts-cantarell,fonts-nanum,fonts-takao-pgothic,gir1.2-totem-plparser-1.0,gnome-session,gnome-themes-standard,gnome-themes-standard-data,gnome-user-share,gpart,gtk2-engines-pixbuf,ipython3,kerneloops-applet,libboost-date-time1.62.0,libboost-filesystem1.62.0,libboost-iostreams1.62.0,libboost-system1.62.0,libboost-thread1.62.0,libcanberra-gtk-module,libcanberra-gtk0,libcapnp-0.5.3,libcdio-cdda1,libcdio-paranoia1,libcdio13,libdfu1,libfreerdp-cache1.1,libfreerdp-client1.1,libfreerdp-codec1.1,libfreerdp-common1.1.0,libfreerdp-core1.1,libfreerdp-crypto1.1,libfreerdp-gdi1.1,libfreerdp-locale1.1,libfreerdp-plugins-standard,libfreerdp-primitives1.1,libfreerdp-utils1.1,libfwupd1,libgom-1.0-common,libicu57,liblouis12,libntfs-3g872,libpython-stdlib,libwinpr-crt0.1,libwinpr-dsparse0.1,libwinpr-environment0.1,libwinpr-file0.1,libwinpr-handle0.1,libwinpr-heap0.1,libwinpr-input0.1,libwinpr-interlocked0.1,libwinpr-library0.1,libwinpr-path0.1,libwinpr-
pool0.1,libwinpr-registry0.1,libwinpr-rpc0.1,libwinpr-sspi0.1,libwinpr-synch0.1,libwinpr-sysinfo0.1,libwinpr-thread0.1,libwinpr-utils0.1,linux-headers-4.13.0-16,linux-headers-4.13.0-16-generic,linux-headers-4.13.0-21,linux-headers-4.13.0-21-generic,linux-headers-4.13.0-38,linux-headers-4.13.0-38-generic,linux-image-4.13.0-16-generic,linux-image-4.13.0-21-generic,linux-image-4.13.0-38-generic,linux-image-extra-4.13.0-16-generic,linux-image-extra-4.13.0-21-generic,linux-image-extra-4.13.0-38-generic,python,python-minimal,python-pexpect,python-ptyprocess,python2.7,python2.7-minimal,python3-decorator,python3-ipdb,python3-ipython,python3-ipython-genutils,python3-pickleshare,python3-prompt-toolkit,python3-pygments,python3-simplegeneric,python3-traitlets,python3-wcwidth,tcpd,ubuntu-gnome-desktop

Some of those packages are in main. I was able to workaround this by setting "allow-insecure=true" in the sources.list file in the apt-clone file, but that is not a solution.

Changed in apt-clone (Ubuntu Bionic):
assignee: nobody → Dimitri John Ledkov (xnox)
Brian Murray (brian-murray) wrote :

I think I was encountering a different problem yesterday as I was trying to ubiquity upgrade an old (not up to date) version of artful. After updating my artful system and then trying the upgrade I'm encountering the error message:

"dpkg: error: cannot stat pathname '/tmp/apt-dpkg-install-7IdNNM': No such file or directory"

This is the one the original reporter experienced.

Brian Murray (brian-murray) wrote :

After modifying apt_clone.py to set "Debug::pkgDPkgPM" I saw the error messages in the attached screenshot.

Brian Murray (brian-murray) wrote :

Another screenshot with different debugging set.

Changed in apt-clone (Ubuntu Bionic):
assignee: Dimitri John Ledkov (xnox) → nobody
Brian Murray (brian-murray) wrote :

Looking around at other ubiquity bugs we can see that apt-clone with the restore-new-distro option has not worked for lots of people for quite some time. As an example bug see bug 1152399 which is a collection of duplicate bugs where the "install" of Ubuntu failed when calculating the upgrade from a previous release to a later release of Ubuntu. This leaves people in a hard to resolve situation and subsequently due to that issue and this crash with apt-clone we have decided to remove the option to upgrade a release of Ubuntu from ubiquity.

Changed in ubiquity (Ubuntu Bionic):
status: New → Triaged
Changed in apt-clone (Ubuntu Bionic):
status: Confirmed → Won't Fix
Changed in ubiquity (Ubuntu Bionic):
importance: Undecided → High
Changed in ubiquity (Ubuntu Bionic):
assignee: nobody → Brian Murray (brian-murray)
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ubiquity - 18.04.6

---------------
ubiquity (18.04.6) bionic; urgency=medium

  [ Steve Langasek ]
  * If we are running ubiquity-dm on VT1, we must conflict with
    <email address hidden> and not rely on some other display manager to do
    this for us.

  [ Brian Murray ]
  * Drop the option to upgrade from an old release using ubiquity as it is
    crashing and leaves users in a broken state. (LP: #1756862)

  [ litianzhi ]
  * Fix untranslated "This software is subject to license" (LP: #1762097)

  [ Mathieu Trudel-Lapierre ]
  * debian/real-po/, po/: Translations import from Launchpad.

 -- Mathieu Trudel-Lapierre <email address hidden> Fri, 13 Apr 2018 15:54:03 -0400

Changed in ubiquity (Ubuntu Bionic):
status: Triaged → Fix Released
summary: - upgrade from 17.10 to 18.04 - apt-clone crashed with SystemError in
- commit(): installArchives() failed
+ UIFE - remove ubiquity upgrade option
description: updated
Brian Murray (brian-murray) wrote :

I've emailed the ubuntu-doc team about this change, https://lists.ubuntu.com/archives/ubuntu-doc/2018-April/020601.html.

Steve Langasek (vorlon) wrote :

Post-hoc approval of the UI Freeze exception, per the understanding that this does not impact the existing docs.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers