Upgrading Ubuntu is risky (unusable or unbootable PC). What can be done to safeguard data, revert failed upgrades and overall reduce the risks?

Bug #876146 reported by manny
126
This bug affects 24 people
Affects Status Importance Assigned to Milestone
NULL Project
Invalid
Undecided
Unassigned
update-manager (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

Please note: This bug report will now be used to track specific code changes for reducing the risk of upgrading (see comment #14). Please stay on topic. We also have enough cases of problematic upgrades. Thank You.

*Report:

===References:

- Typical bad upgrade user case (forum post):
http://ubuntuforums.org/showthread.php?t=1860359

- Unusable upgrade pic (unbootable)
http://ubuntuforums.org/attachment.php?attachmentid=204320&d=1318627364

- [ubuntu] Thread for failed upgrades (huge list within 24 to 48 hours)
http://ubuntuforums.org/showthread.php?t=1858475

- Unofficial poll, but might serve useful as another source of data and user cases. Also, reference for future polls. Current sample of about 6k users, ~22% needed to reinstall.

http://www.reddit.com/r/Ubuntu/comments/lm036/how_well_did_your_ubuntu_1110_upgrade_go/

11.04 to 11.10 (26% failed.):
http://www.omgubuntu.co.uk/2011/10/how-well-did-your-ubuntu-11-10-upgrade-go/

11.10 to 12.04 LTS (18% failed, about ~8% improvement):
http://www.omgubuntu.co.uk/2012/05/how-will-did-your-ubuntu-12-04-upgrade-go-poll/

===Description:

Lots of angry users with broken computers that need rescue...

The friendly Upgrade popup has become a mouse trap for new and unsuspecting users.

Warnings and fail-safe alternatives and/or advise should be offered. Specially for users who cant risk having their computers broken (only have 1 OS). Nor have the knowledge, will, time and patience to fix something like this.

I think this should be Critical for "Precise" pangolin as the number of users upgrading will be in the Tens if not hundreds of thousands (or millions...), since this will be the new LTS.

===Initial proposal:

See comment #10:
https://bugs.launchpad.net/ayatana-design/+bug/876146/comments/10

===Other related helpers for the upgrade problem:

-There is no "feedback and reporting tool to count the number of bad upgrades" and their related causes. Users should be able to give immediate feedback after every upgrade (successful or not). "Ubuntu-bug upgrade-report" on startup after an upgrade and a section on launchpad or dedicated site.

-Update/upgrade manager should offer an auto-backup wizard to safe guard important user files to another partition, external storage or online (See comment #10).

-Also separate branches for testing and a better release process/cycle would certainly help diminish the amount of bugs and many upgrade problems as it has been discussed here:
http://netsplit.com/2011/09/08/new-ubuntu-release-process/

Revision history for this message
manny (estelar57) wrote :

sorry, while the problem is indeed critical, i think it was marked as security vulnerability but am not sure it is, how to unmark it?

manny (estelar57)
description: updated
Revision history for this message
manny (estelar57) wrote :

And here is a testimonial from a volunteer helper (Effenberg) in the ubuntuforums.

http://ubuntuforums.org/showpost.php?p=11339157&postcount=30

"I spent a couple hours on the "General Help" forum now. I answered posts till I had to stop and smoke an entire pack. Many users ARE having problems with broken / frozen installs that lead to "No OS" BIOS messages or no possible boot on grub. They have systems stuck in purple screens (plymouth + no/wrong video setup), can't login to lightdm (wrong user session env vars, ~/.Xauthority, among others), login to desktop with no launcher/dash (unity/compiz common failure).

Looking at my stats, I see that I answered more than 60 posts like this in one afternoon

Trying to help them, I realized that they can't use the terminal, switch to a VT, cut/paste the commands I wrote, answer what VGA they have, perform sequences of commands as advised to, they can't install the correct VGA drivers and its impossible for them to perform adjustments (env vars, settings in ccsm, etc) to work around VGA bugs, etc.

How many times were these things reported? I know developers are not gods, of course. But I still feel like my dream would that the many brains behind development focused less on interface and more on fixing these age-old bugs / making the OS truly easy to install and use, bulletproof.

I really like Unity, but today, for the first time, I feel like it was too big a step. Average users won't even see Unity. They're trying to get to any desktop...

Regards,
Effenberg"

I would like to thank people like "Effenberg" for their support.

security vulnerability: yes → no
visibility: private → public
Revision history for this message
Matthew Paul Thomas (mpt) wrote :

manny, thanks for this report. I understand your frustration and that of others on the forums.

To address your points in order:

<http://ubuntuforums.org/showthread.php?t=1860359> obviously illustrates a bug, but the thread does not yet contain any information that a developer could use to identify and fix the bug. If any of the threads linked from <http://ubuntuforums.org/showthread.php?t=1858475> represent a reproducible bug, please do report it individually so it can be fixed.

If upgrading works only some of the time, warning the user is not a reasonable solution -- because it would be asking them to make a decision that they can't possibly have enough information to make. (Will it work on my machine? Nobody knows!) There are two practical solutions: make upgrading more reliable, or not offer the function at all.

Future Ubuntu releases will likely allow automated feedback on whether upgrades succeeded. That will be much more representative, and useful, than manual reporting.

And finally, if any changes are made to the release cycle, they are unlikely to themselves be tracked in bug reports.

For future reference, the Ayatana Design project is a dummy project used by the designers of Unity specifically. It is not for process problems or bugs in packages outside of Unity.

Changed in ayatana-design:
status: New → Invalid
Revision history for this message
manny (estelar57) wrote :

@Matthew Paul Thomas (mpt)

Thanks for the feedback, i was not sure where to place this as it affects many areas in ubuntu.

Can anyone suggest specific areas or does this affects ubuntu itself?

Also the "Upgrade Popup" (not sure which project it is) needs some changes in design to offer some alternatives that are safer for users who just want to try a new release, but keep their old/current one intact and functional (specially if this is their only OS installed).

Sure, is not intentional, but right now is like a mouse trap waiting for a new unsuspecting victim to click on the button...

We do everything we can to help in the forums, but users tolerance are getting thinner with every release.

For example this next user had several bad updates/upgrades and even lost data, some other users instead of helping were trying to fault him and things went down hill pretty fast. The sad thing is that these type of cases are not uncommon:
http://ubuntuforums.org/showthread.php?t=1861749

Many more go unreported and they just switch back to what they were using before or go elsewhere.

Upgrading or not users need a warranty that their computers will always be bootable or have some sort of fall back all the time.

I agree with both practical solutions: "make upgrading more reliable, or not offer the function at all."

Since upgrading is risky, this should be left only to advance users who are not afraid of breaking their system, because they have multiple fall backs (various computers or various test partitions) and have the patience/knowledge/time to fix the problems.

***So the initial proposal would be something like this:

- Dont show the upgrade button to users. Not "directly" upgrading is the only warranty that non technical users will have a working system.

- The "Upgrade popup" should become a "New release available to try" popup: this means offering fail-safe options like try from an USB stick or CD (clicking the TRY button should automatically download the latest ISO and let the user choose create a live- USB or CD).

- The commands should still be available, but only to advance users who research them first. Those who want to take this path should also get a warning that its at their own risk and that Canonical is not responsible. But they will be able to send feedback or upload an upgrade-report to improve the process (like you mentioned).

- In the future when upgrading becomes much more reliable (and thanks to the creation of backup first, integration with Btrfs's snapshot system or something else), then make the function 1 click again.

The new PP cycle should be specially important for keeping ubuntu as problem free as possible.

Thank you.

manny (estelar57)
description: updated
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ubuntu:
status: New → Confirmed
Revision history for this message
Mike Ferreira (mafoelffen) wrote :

Offhand the past 5 days helping user with what broke has been adventurous...

I think the main update issue lies in the dist-upgarde process. In the last 3 dev testing cycles, one thing has been nagging at ne is that there is no prior testing to see if an existing previous version will successfully do a distribution upgrade to the pre-release. That would see before hand what things would break right? Instead, we just did it.

Along side of this is new installs. There is existing bugs and issues graphically with the installer, that we tell users to correct manually. Can we not add some kind of logic to sort some of these issues out? It already probes hardware to configure an install. It already has a user selected 3rd party button to configure proprietary software. Could it not configure appropriate graphics drivers, so that the first reboot doesn't end at the now infamous blackscreen?

Revision history for this message
Tom Pino (metalsmith-rangeweb) wrote :

This problem has been getting worse with each release for several releases now. It is now getting to be too much of a problem to ignore.

A warning, at least, would be a good start.

Getting more upgrade testing, as was done for the 8.04 to 10.04 upgrade should really become standard for all releases.

I know it is harder to get testers for the regular releases but there must be some way to encourage more.

Revision history for this message
Matthew Paul Thomas (mpt) wrote :

So the claim here is that upgrading is dangerous enough that people need to be warned against it.

I submit that nobody knows whether it is dangerous or not. Nobody has ever measured the success rate for upgrading to any version of Ubuntu. We have very little idea whether the success rate is high or low, or whether it's improving or getting worse.

It would be easy to assume that more problems posted on the forums means it's getting worse. But so far, each Ubuntu release has been installed by more people than the previous one. So it might be that the success rate is staying constant or even improving, while the number of failures reported on the forums still increases.

affects: ayatana-design → null
Revision history for this message
manny (estelar57) wrote :

Just wanted to add, that linux is extremely flexible in alternatives that can be offered to the user for fall-back (more so than any other OS out there). Imagination can go wild here.

-The user wants a full snapshot of entire system? we have clonezilla and other tools (even for creating live-cds of user defaults).

-The user wants upgrade manager to create a temporary backup partition to revert back in case of problem? it can be done

-User wants the system to auto download and create a bootable usb or cd? yes it can! and unetbootin is also an option

-User wants a restore partition (like what OEMs do with windows), it can be done.

-the 1 click "no questions asked" upgrade button, can even become an upgrade survey/wizard, determine your level of technical experience and offer alternatives based on that: Are you a technical user with other fall back OSs? great break away! ; Are you a granny with no prior technical experience? sorry upgrade denied, but use these methods instead to try a new release!

We have not offer new users more alternatives, not because it cant be done or they don't exist, but because we havent taken a few hours to think and discuss the situation, nor we have gathered the feedback.

Instead we offer the only alternative that is known to cause potential problems: a risky direct upgrade on the user's primary OS.

Today, the operation is supposedly offered for "convenience". But when the user faces critical issues, is far from being convenient anymore and turns into a mistake he wishes to revert back.

This was a popular legacy operation that had to be used due to serious limitations of disk space and internet bandwidth.

The OS use to take 50% of the hard drive, but now with HDDs reaching 1TB mark, they only take a very small % of the free space and even users with smaller drives have plenty of space for backups and other external medias. We dont have as many limitations as before.

Hopefully this topic will also see some light at the UDS. This is a "Precise" timing :)

>So it might be that the success rate is staying constant or even improving, while the number of failures reported on the forums still increases.

@Matthew Paul Thomas (mpt)

Yea that could be the case too, but the only way to measure that would be to add the feature into "Apport" (or other tool) and send the feedback to either launchpad or another dedicated websites.

Still the bad upgrade cases still Exists (they are not made up), and even if its just 1% of our current base suffer any kind of bad upgrade, from medium to severe (1% of 12m, thats equal to 120,000 bad cases; imagine if we reach anything near the proposed 200 million...)

So providing in the meantime some fall backs for non tech users will avoid quite a few headaches and downtime.

Thanks!

Revision history for this message
manny (estelar57) wrote :

@Matthew (mpt)

Hi, it was recently brought to my attention from a few mac users, that Osx 10.7 lion comes with a new and improved auto-backup tool prior to upgrading.

"The first thing the 10.7 installer tells you is to backup your system and even if you do, it makes you a recovery partition."

Some reference links with details and pics:

http://www.macworld.com/article/161664/2011/08/hands_on_with_lion_recovery_disk_assistant.html
http://reviews.cnet.co.uk/software-and-web-apps/how-to-recover-os-x-lion-50004503/
http://maciad.com/news/os-x-lion-recovery-detailed
http://osxdaily.com/2011/07/20/upgrading-to-mac-os-x-10-7-lion/
http://drive-recovery-mac.blogspot.com/2011/08/lion-recovery-disk-assistant.html
http://www.sinfuliphone.com/showthread.php?t=85170
http://www.macgasm.net/2011/06/13/reinstall-lion-disc/

Ubuntu is still behind in this area...

But just recently we saw the inclusion of a very good and easy to use back up tool: Dejadup.

More info:
http://live.gnome.org/DejaDup

Dejadup is even looking to propose installer integration:
http://live.gnome.org/DejaDup/Design/Proposal-InstallerIntegration

Is time to take advantage of it and continue with plans of further integration with other important components like the "upgrade/update manager".

While aiming for a 0% downtime policy should be the goal, Data protection and easy recovery in case of failures or upgrade problems should also become top priority.

So to sum the needs that have been identified up till now:

-Auto feedback and upgrade report tool (Apport team?)

-Upgrade Backup Assistant similar to osx's tool (Dejadup and Upgrade-manager team to handle integration details).

-Installer integration of Dejadup also ?

-Advise users of alternatives for "testing" new releases first (like live-usb or cd).

description: updated
Revision history for this message
Matthew Paul Thomas (mpt) wrote :

Okay, there are several good suggestions there, which would be useful no matter how reliable upgrading actually is. Thank you.

affects: ubuntu → update-manager (Ubuntu)
Revision history for this message
Subharo Bhikkhu (subharo) wrote :
Download full text (4.3 KiB)

I have been using some kind of linux distro since 1997, and in all that time, I have never, ever been "done like this" before. I have a few specific egregious gripes to add, after dragging myself out of a horrendous upgrade. Good thing I have a Computer Science Degree, and 5 years experience as a Unix System Administrator. I pity the newbies. I'll never stray from an LTS-to-LTS upgrade again.

Before I get into specific, difficult-to-believe gripes, Mark Shuttleworth, I beseech you: PLEASE BASE UBUNTU OFF DEBIAN STABLE, NOT DEBIAN TESTING. Why? Because Ubuntu is CUTTING EDGE ENOUGH now, and has all the basic application functionality well-covered that 98% of all users will want. No need to skate the edge anymore. Quit grinding us. If you want this to be "Linux for Human beings", then make it stable, and keep it stable. Stability should come ahead of being cutting edge. The cutting edge is inherently NOT for the masses. That's an axiom you can take to the bank.

OK, here are some specific, huge upgrading problems, described in general terms. I provide them mainly to underscore my general point above. No, I don't have the time to open like 15 seperate bug reports for each of these.

-When the upgrade popup first appeared inviting me to upgrade to 11.10, when I pressed the "Remind me later" button, it did nothing. I had to press the X in the upper right to make the invitation go away. It makes we wonder, "Is that installer tool QA'ed whatsoever? How did they miss that button being broken, which about 90% of all users will surely click (who don't want to upgrade that very instant they are first invited)."

-I wanted to use the "alternate" installation CD, to cut down on bandwidth used. This is because I'm on satellite internet, where the bandwidth is metered, and precious. Once this install was initiated, the installer asked if I wanted to download newer packages in addition to installing from CD. By pressing the "No," button it promised to not download anything from the internet, and just install from the CD. When I pressed "No", the installer DOWNLOADS FROM THE INTERNET ANYWAY.

-To confirm that I wasn't hallucinating, I wanted to open System Monitor to see if in fact downloading was occuring. The very instant I launched it, my DESKTOP SESSION WENT ALL WONKY. The sidebar along the left DISAPPEARED. Only two icons on the desktop remained, with my desktop background still there. But they flashed in and out of existence,and were unresponsive. The installer also flashed in and out of existence, and lost all it's window decorations! The installer froze when it wanted to make a popup to ask me a question, but that popup wouldn't show up at all. (At the time of configuring libpam0g). I had to kill the X session and try to continue the install, in a now-broken state from the command line.

-I discovered the alternate install CD was having I/O errors, even though it was freshly burnt (good thing I knew to look in the logs), since THE INSTALLER MADE NO CHECKSUM CHECK BEFORE INSTALLING, NOR INFORMED ME OF THE I/O ERRORS. Good thing I knew how to remove the CD from /etc/apt/sources.list, and "sudo apt-get update".

-After mu...

Read more...

Revision history for this message
manny (estelar57) wrote :

@Subharo Bhikkhu (subharo)

Thanks for sharing your experience. Sorry for your frustrations, as It seems you encountered a great number of bugs, a bad upgrade and other issues..

Scott James blogged about the state of Ubuntu, (check the link in the bug report at the top).

And Matthew (mpt)'s comment hit the nail too:
http://netsplit.com/2011/09/08/new-ubuntu-release-process/#comment-2966

"Scott, I’m amused at how we’ve identified exactly the same problems with the Ubuntu development process (I mentioned some of them in my plenary talk last UDS), but we’ve come up with diametrically opposed solutions. You think we should have a release possibly containing new features every month. I think we should have one every two years.

We agree on the root cause: Ubuntu has not just time-based releases, but time-based features. Every problem is assumed to have a solution that can be implemented either in 13 weeks, or in landable 13-week chunks. Often that just isn’t true, and in trying to make it true, we release half-baked designs or implementations or both. (Or in the worst case, nobody bothers even starting on the feature.)

Moving to an always-stable-trunk model, as you describe, is something that could work no matter how frequent the releases were [.....]"

@mpt
Am not sure but is this the "Thursday" plenary O talk you mentioned (future of universe)?
http://www.youtube.com/watch?v=-oWN9qLwEkM

or is it another one?

The problems seem to be identified, but am not sure if everyone is willing to cooperate to find a common solution?

Revision history for this message
Matthew Paul Thomas (mpt) wrote :

Please do not use this bug report to post anecdotal problems with upgrading or to suggest changes in the release cycle. This bug report will track specific code changes for reducing the risk of upgrading. If it stops being useful for that purpose, I will mark it Invalid.

Revision history for this message
manny (estelar57) wrote :

@Matthew Paul Thomas (mpt)

Sorry this was not intentional.

Just wanted to refer to @Subharo that changing ubuntu to a different base (like debian stable) may not solve the root problem.

But you are right, this is not the correct bug report to discuss this as it can start drifting off topic. As you mentioned It should only be used to track the specific changes for reducing the upgrading risks.

manny (estelar57)
description: updated
manny (estelar57)
description: updated
Revision history for this message
Jeremy Bícha (jbicha) wrote :

There is a UDS session scheduled for next hour (noon EDT - GMT-4 ) where I'd like to discuss what we can do to improve upgrades for 12.04. Some ideas will of course be too big to land in this release cycle so we want to focus on the smaller improvements that can really make a difference in the very short term.

https://blueprints.launchpad.net/ubuntu/+spec/desktop-p-improve-upgrade-experience
http://summit.ubuntu.com/uds-p/meeting/19831/desktop-p-improve-upgrade-experience/
http://uds.ubuntu.com/participate/remote/

manny (estelar57)
summary: - Upgrading Ubuntu is risky (unusable or unbootable PC). The Upgrade Popup
- does not warn of the risks or offers fail-safe alternatives. This is a
- mouse trap for unsuspecting users.
+ Upgrading Ubuntu is risky (unusable or unbootable PC). What can be done
+ to safeguard data, revert failed upgrades and overall reduce the risks?
Revision history for this message
Matthew Paul Thomas (mpt) wrote :

I've talked with several people over the past couple of days about different ways of restoring a system after a failed upgrade. These include:

* Using Déjà Dup to back up files before upgrading. Difficulties: (1) Déjà Dup currently targets your home folder, not system files; (2) a full incremental backup system is overload for this fairly specific task; (3) no-one has time to do the work.

* Inviting people to back up their system files to an external disk or USB key. Difficulties: (1) it's not technically obvious how the restore function would work; (2) no-one has time to do the work.

* Using the existing friendly-recovery software. Difficulties: (1) It's console-based, so it has a hard friendliness limit; (2) it couldn't be graphical, because if it was, it would be using your disk too much to be able to fix errors on it; (3) no-one has time to do further work on it this cycle.

* Installing a recovery image on a separate partition during installation. Many Ubuntu OEM installations already do this <https://launchpad.net/ubuntu-recovery>, but to help users of vanilla Ubuntu, the setup would need to be integrated into Ubiquity. Difficulties: (1) while it would help people installing from scratch, it wouldn't help upgraders, because during an upgrade, the ext4 partition you're using right now can't easily be resized to make room for the separate partition; (2) no-one has time to do the work.

* Installing a recovery image as a file on the same partition as the installation. Grub can happily boot from ISO files inside partitions. Difficulties: (1) it wouldn't help if the partition gets hosed; (2) no-one has time to do the work.

Now, when I say "no-one has time to do the work", I'm referring to the people I've talked to about it. If anyone has a good idea about how they can implement any of these options, I'd be happy to sketch out a design for them to work on. Until then, though, I'm going to postpone design work on this in favor of things that I know people plan to implement.

Revision history for this message
Matthew Paul Thomas (mpt) wrote :

Oh, I forgot one:

* Btrfs mumble mumble handwave mumble mumble mumble snapshots mumble rollbacks. Difficulties: (1) Btrfs still isn't ready to be the default filesystem in Ubuntu (for example, it still doesn't have a released fsck); (2) no-one has time to do the work; (3) Btrfs imposes a chilling effect on people who otherwise might work on any of the other solutions.

Revision history for this message
manny (estelar57) wrote :

Oh wow. Everyone seems to be virtually at full capacity, but since this is still very important I think that the implementation will need to be "As Simple As Possible" for now and then refine or automate it more later when there's a little more resources available.

Well I believe the most important thing to remind users is for them to backup the "files that matter to them". Be it to a External-disk or Online (U1).

Their stuff is now safeguarded in case that something unexpected happens.

As for restoring, I don't see a problem of doing it manually for now. And usually when people restore they dont copy everything over since they see the backup as another safe place to keep a lot of their stuff.

As for restoring the Apps i think USC has a sync feature ? But Am not very familiar with that implementation yet.

Revision history for this message
manny (estelar57) wrote :

Apart from recommending the user to backup the files that matter to them.

Another thing that is important and should be recommended is for the user to try/test the new ubuntu version first.

And this can only be done right now with a disc or usb stick.

So if you insert a disc/usb one can be automatically created for you.

Once the user has tested the new version and verified everything works and the performance is still acceptable on their hardware, then they can choose to install/upgrade, right there from the disc/stick.

I think testing it first avoids some headaches and wasted-time by having to revert everything back to a previous state/version later on.

Sure, You can still upgrade without doing the recommended steps of "testing and backing up", but do so at your risk, only if you know what you're doing and have experienced to fix potential problems yourself.

Revision history for this message
Matthias Kretz (mkretz) wrote :

Hi, question: How does the upgrade QA work? Does it rely solely on users that do the upgrade at Alpha and Beta stages? Do you have a simple way to repeat upgrade processes to test the progress of stabilization?

Actually I'd be happy to do upgrade tests (especially since my experience is that something always breaks). This scenario could work for me:
* Alpha/Beta is announced
* When I know I won't need my computer for the next few hours I will tell it to do a test upgrade
* The upgrade process creates a bootable USB key or CD to restore the system later
* First it tests whether this restore system boots, only if this works does the upgrade proceed
* After the upgrade is done I can check for problems and report them (this must be automated as much as possible)
* When done, I reboot into the restore CD/USB and get my old system back
* If I get notified that some of my issues should be fixed I can repeat the process to verify

Except for bootloader issues this should be easily possible given enough space on the harddrive...

PS: I installed Ubuntu for a friend and when the popup appeared that a distribution upgrade is available, the first thing I did was call my friend and tell him to not click that button - I was just in time to stop him (and I'm happy I did - two out of two Oneiric upgrades broke for me. I couldn't reserve enough time yet for the remaining computers that want to upgrade.).

Revision history for this message
Walter Garcia-Fontes (walter-garcia) wrote :

I have duplicacted here bug 613174 , just in case space checking for the upgrade to a new release can be improved, since under some circumstances it may abort the upgrade and leave the system in a bad state.

Revision history for this message
manny (estelar57) wrote :
manny (estelar57)
description: updated
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Related blueprints

Remote bug watches

Bug watches keep track of this bug in other bug trackers.