Unattended upgrades can break persistent live media

Bug #1619188 reported by sudodus on 2016-09-01
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
casper (Ubuntu)
Medium
Brian Murray
Xenial
Medium
Brian Murray
Yakkety
Medium
Brian Murray

Bug Description

Test Case
---------
1) Boot an Ubuntu 16.04 Live CD
2) Choose Try Ubuntu
3) head -n5 /etc/apt/apt.conf.d/50unattended-upgrades
4) Observe -security is enabled in line 3. (// is a comment)
5) Run /usr/lib/apt/apt.systemd.daily
6) Observe your Live environment run out of space! (I received a pop-up re lack of free space and /var/log/unattended-upgrades/unattended-upgrades-dpkg.log contained an error installing a package due to disk full.)

With the version of the package in -proposed step 4 will reveal the -security pocket being disabled and step 6 won't fill your live environment.

Regression Potential
--------------------
Persistent live users will not receive updates from -security, but that seems less bad than destroying people's live environment by filling up their disk.

Original Bug Description
------------------------
Looking at the persistent live Ubuntu 16.04 LTS system - the Software & Updates screen / Update - I notice, that Automatic updates is set to 'Download and install automatically'. This is bad in a persistent live system.

After leaving the persistent live Ubuntu 16.04 LTS system running overnight, I found that it had performed an automatic upgrade:

df revealed that the content in casper-rw had increased to 1.6 GiB.

It was a surprise that the persistent live system started an automatic upgrade. This is not caused by the installer (mkusb), because the files controlling those actions are not touched. Instead it is caused by a change of the default action, when there are security updates. And the survey indicates that this default setting is different between the flavours and versions of Ubuntu. Lubuntu keeps the setting 'Display immediately', while the other tested flavours change it from 14.04 LTS to 16.04 LTS.

Until this is resolved, it is a good idea to disable unattended-upgrades manually, but above all, to take regular backups, when you use a persistent live system.

The survey is described in this link to the Ubuntu Forums:

https://ubuntuforums.org/showthread.php?t=2335669&p=13538805#post13538805

and it contains some screenshots illustrating the settings manager for automatic updates for different versions and flavours of Ubuntu.

-o-

There are various scripts in the casper package that change things to be more appropriate for the live environment. I suggest to change the default for security updates to 'Display immediately', when the system is running live and persistent live.

See also this link to Ask Ubuntu, the first report about this problem:

https://askubuntu.com/questions/817750/unattended-upgrades-broke-persistent-live-media/817820#

-o-

Addendum: Things should continue to work if you leave a live Ubuntu iso running for a day or two. Unattended upgrades cause problems for all live systems, 'live-only' and 'persistent live'.

ProblemType: BugDistroRelease: Ubuntu 16.04
Package: casper 1.376
ProcVersionSignature: Ubuntu 4.4.0-31.50-generic 4.4.13
Uname: Linux 4.4.0-31-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
CasperVersion: 1.376
CurrentDesktop: Unity
Date: Thu Sep 1 08:32:08 2016
LiveMediaBuild: Ubuntu 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bashSourcePackage: casper
UpgradeStatus: No upgrade log present (probably fresh install)

sudodus (nio-wiklund) wrote :
sudodus (nio-wiklund) wrote :

- This bug affects 16.04 LTS, 16.04.1 LTS and Yakkety.

- Several flavours are affected.

- Lubuntu is not affected.

One screenshot is attached here. You find more screenshots at

https://ubuntuforums.org/showthread.php?t=2335669&p=13538805#post13538805

sudodus (nio-wiklund) on 2016-09-01
description: updated
Ubuntu QA Website (ubuntuqa) wrote :

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1619188

tags: added: iso-testing
sudodus (nio-wiklund) wrote :

Would these changes be enough to change the setting from 'Download and install automatically' to 'Display immediately' in Ubuntu 16.04.1 LTS?

-----
ubuntu@ubuntu:/etc/apt/apt.conf.d$ for i in *;do diff $i orig || echo "^^^ diff in $i";done
4d3
< APT::Periodic::Unattended-Upgrade "0";
^^^ diff in 10periodic
2,4c2
< APT::Periodic::Download-Upgradeable-Packages "0";
< APT::Periodic::AutocleanInterval "0";
< APT::Periodic::Unattended-Upgrade "0";
---
> APT::Periodic::Unattended-Upgrade "1";
^^^ diff in 20auto-upgrades
ubuntu@ubuntu:/etc/apt/apt.conf.d$
-----
See also the attached screenshot?

Are these updating settings saved somewhere else in other versions?

Are there other files that would need modifications too?

tags: added: rls-y-incoming
sudodus (nio-wiklund) wrote :

I understand that the following format is better for patching: diff -u old-file newfile

-----
ubuntu@ubuntu:/etc/apt/apt.conf.d$ for i in *;do diff -u orig/$i .;done
--- orig/10periodic 2016-05-24 16:48:53.000000000 +0000
+++ ./10periodic 2016-09-01 16:56:26.303618989 +0000
@@ -1,3 +1,4 @@
 APT::Periodic::Update-Package-Lists "1";
 APT::Periodic::Download-Upgradeable-Packages "0";
 APT::Periodic::AutocleanInterval "0";
+APT::Periodic::Unattended-Upgrade "0";
--- orig/20auto-upgrades 2016-02-18 22:19:18.000000000 +0000
+++ ./20auto-upgrades 2016-09-01 16:56:26.303618989 +0000
@@ -1,2 +1,4 @@
 APT::Periodic::Update-Package-Lists "1";
-APT::Periodic::Unattended-Upgrade "1";
+APT::Periodic::Download-Upgradeable-Packages "0";
+APT::Periodic::AutocleanInterval "0";
+APT::Periodic::Unattended-Upgrade "0";
diff: orig/orig: No such file or directory
ubuntu@ubuntu:/etc/apt/apt.conf.d$
-----

So this should be better in order to create a real patch.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in casper (Ubuntu):
status: New → Confirmed

Because most inexpensive modern USB drives will be able to handle that amount of space without problems, I believe changing the default upgrade method would be a drawback.

For example a Kingston 32GB USB 3.0 drive costs as little as 8€, and a Sandisk 8GB USB 2.0 drive costs 3€. The official Ubuntu USB drive has 32GB and costs 6£.

Upgrading automatically is a better default for at least 95% of situations, and the person could still easily realise that they should disable automatic upgrades when having a small vintage USB drive. Moreover the Startup Disk Creator no longer has persistence, so the likelihood of a novel user making that mistake is minimum.

For the moment I will consider this report as an opinion, but feel free to bring the conversation to the Quality mailing list at (https://launchpad.net/~ubuntu-testing). Thanks for your always welcomed help and understanding.

Changed in casper (Ubuntu):
status: Confirmed → Opinion
importance: Undecided → Wishlist

Den 2016-09-02 kl. 18:21, skrev Alberto Salvia Novella:
> Because most inexpensive modern USB drives will be able to handle that
> amount of space without problems, I believe changing the default upgrade
> method would be a drawback.
>
> For example a Kingston 32GB USB 3.0 drive costs as little as 8€, and a
> Sandisk 8GB USB 2.0 drive costs 3€. The official Ubuntu USB drive has
> 32GB and costs 6£.
>
> Upgrading automatically is a better default for at least 95% of
> situations, and the person could still easily realise that they should
> disable automatic upgrades when having a small vintage USB drive.
> Moreover the Startup Disk Creator no longer has persistence, so the
> likelihood of a novel user making that mistake is minimum.
>
> For the moment I will consider this report as an opinion, but feel free
> to bring the conversation to the Quality mailing list at
> (https://launchpad.net/~ubuntu-testing). Thanks for your always welcomed
> help and understanding.
>
> ** Changed in: casper (Ubuntu)
> Status: Confirmed => Opinion
>
> ** Changed in: casper (Ubuntu)
> Importance: Undecided => Wishlist
>

Hi Alberto,

I see your point, but there is definitely a problem in this case, a
problem big enough to report a bug.

1. Several tools for persistence are still creating a casper-rw file in
a FAT32 file system. This will limit the size of the file and the
loop-mounted filesystem inside it to 4 GB, which is definitely not
enough, if you allow unattended upgrading.

2. mkusb is different, it creates a partition for persistence, but I am
sure that many people want to use 4 GB or 8 GB pendrives, and there is
definitely a space problem also in these cases.

3. But maybe the most severe reason to avoid excessive upgrading is that
persistent live systems are much more sensitive than normal installed
systems. It is well known to people who use persistent live systems,
that you should keep the amount of upgrading to a minimum.

The kernel and certain other parts of the system cannot be upgraded,
because the kernel from the live system is started before the overlay
structure is created from the casper-rw file or partition. Some upgrades
that will be used imply that the new kernel etc are used, but that is
not possible, so the system fails to work. As it is now, the kernel is
upgraded (but not used).

-o-

I have discussed this issue with two very knowledgeable persons with own
experience of persistent live systems, and we agree: Changing the
default setting of updating program packages (as what happened from
Ubuntu 14.04 LTS to 16.04 LTS) creates problems for persistent live
systems - problems that are severe enough to say it is a bug.

Many disappointed users will probably leave Ubuntu silently and select
some other distro after a system crash because of this bug.

-o-

Please notice that I want to keep the general settings of the updating
system [of installed systems], and only suggest changing the settings,
when running live and persistent live. This is why the bug is directed
against casper.

Best regards
Nio

In that scenario, changing the setting when running a live environment sounds good to me.

Changed in casper (Ubuntu):
importance: Wishlist → Medium
status: Opinion → Confirmed

What if the security updates were the only enabled by default, and those were installed automatically?

Jeremy Bicha (jbicha) wrote :

I think unattended upgrades should be disabled by default on the live media since there isn't space for upgrades when the Ubuntu live images is run 1) from a DVD or 2) from the no-persistence that recent versions of usb-creator-gtk unconditionally uses or 3) as written by dd. I believe it's similarly inappropriate even to notify about potential updates even security ones in this environment.

sudodus, if you're interested in trying to submit a patch for this, look at the scripts/casper-bottom/ directory (in the casper source) which is where these kinds of customizations of the live environment are made.

Jeremy Bicha (jbicha) wrote :

Things shouldn't break if you leave an Ubuntu iso running for a day or two.

sudodus (nio-wiklund) wrote :

You mean similar to what is done in Linux Mint, where there are different priority levels for the upgrades? It may help, but would be more difficult to implement, and can still cause problems, when the security upgrades accumulate.

I do not say that people should ignore the need for security upgrades. I think it is important to get a 'heads up' warning as soon as possible. This is done with the alternative 'Display immediately', which is the default in Ubuntu 14.04.x LTS.

Persistent live systems are sensitive, and it is important for the user to control, if, how and when to perform upgrades.

If the user needs higher security, I think it is better to use an installed system or a live or persistent live system made from the current daily iso file as described in the following link:

https://askubuntu.com/questions/817750/unattended-upgrades-broke-persistent-live-media

Quote: 'or grab the current 16.04 LTS daily iso file, and create a new persistent live system. You find it via this link: http://iso.qa.ubuntu.com/qatracker/milestones/351/builds. In your case, select a version of Mythbuntu, and you will find a link to the download information. (The 16.04 daily iso files will no longer be updated after 16.04.5 is released.)'

sudodus (nio-wiklund) wrote :

[The previous comment was a reply to Alberto]

Jeremy, I think you are right. This is important not only for persistent live systems, but to live-only systems too.

Jarno Suni (jarnos) wrote :

Alberto, just security updates are enabled by default now in most flavours. It is configured in /etc/apt/apt.conf.d/50unattended-upgrades

sudodus (nio-wiklund) on 2016-09-03
description: updated
description: updated
Jarno Suni (jarnos) wrote :

Do you have experience on how unattended upgrades behave in live-only systems? The permanent file system is mounted read-only, I suppose, so only RAM and maybe swap is available for writing, right?

sudodus (nio-wiklund) wrote :

Right!

I have never left a live xenial or yakkety system running overnight. I think, that it will also try to run the security upgrades, if you leave it overnight, but it is worth testing. A live system might survive one or more security upgrades with 4 GB RAM plus swap, but it would be a terrible waste of RAM and effort. I think it would be rather easy to make it crash :-P

Fortunately Lubuntu has kept the default 'Display immediately', so people with old computers and low amount of RAM, who use Lubuntu, are not affected.

There is a solution via mkusb version 11.0.2 in the pipeline. You can test it via ppa:mkusb/unstable, but the real solution is to modify Ubuntu (according to this bug report).

I'm unsubscribing from this report now. Feel free to subscribe me again if you have any further question.

sudodus (nio-wiklund) wrote :

I tested a live only session with Ubuntu 16.04.1 LTS amd64 last night and it did an unattended upgrade, and after that it used 726 MiB of the allocated space for the root file system:

ubuntu@ubuntu:~$ df -h .
Filesystem Size Used Avail Use% Mounted on
/cow 1.9G 21M 1.9G 2% /
ubuntu@ubuntu:~$ df -h .
Filesystem Size Used Avail Use% Mounted on
/cow 1.9G 726M 1.2G 38% /
ubuntu@ubuntu:~$ df -hi .
Filesystem Inodes IUsed IFree IUse% Mounted on
/cow 480K 37K 443K 8% /
ubuntu@ubuntu:~$ uname -a
Linux ubuntu 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@ubuntu:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.1 LTS
Release: 16.04
Codename: xenial
ubuntu@ubuntu:~$

tags: removed: rls-y-incoming
Changed in casper (Ubuntu Yakkety):
status: Confirmed → Triaged
Brian Murray (brian-murray) wrote :

It's trivial to recreate this by running /usr/lib/apt/apt.system.daily, the script called by /etc/cron.daily/apt-compat, on a Live CD.

Just disabling the -security origin in /etc/apt/apt.conf.d/50unattended-upgrades will fix it.

Changed in casper (Ubuntu Yakkety):
assignee: nobody → Brian Murray (brian-murray)
status: Triaged → In Progress
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package casper - 1.379

---------------
casper (1.379) yakkety; urgency=medium

  * scripts/casper-bottom/53disable_unattended_upgrades:
    - Disable the security allowed origin as it can fill a live environment.
      (LP: #1619188)

 -- Brian Murray <email address hidden> Fri, 30 Sep 2016 13:57:37 -0700

Changed in casper (Ubuntu Yakkety):
status: In Progress → Fix Released
sudodus (nio-wiklund) wrote :

I'm looking forward to test it, when it arrives in the daily iso file :-)

Brian Murray (brian-murray) wrote :

I tested this using a Yakkety image from 20161002 which included casper 1.379. The file /etc/apt/apt.conf.d/50unattended-upgrades had the -security pocket commented out, and running /usr/lib/apt/apt.systemd.daily did not kill the system.

I'd appreciate it if other people tested too, just to be sure.

Brian Murray (brian-murray) wrote :

After installing the system the -security pocket was enabled in /etc/apt/apt.conf.d/50unattended-upgrades.

Changed in casper (Ubuntu Xenial):
status: New → Triaged
importance: Undecided → Medium
assignee: nobody → Brian Murray (brian-murray)
Changed in casper (Ubuntu Xenial):
status: Triaged → In Progress
sudodus (nio-wiklund) wrote :

I will start the current daily Ubuntu (released Oct 3), and run it live (without persistence) overnight. I hope it works, but it makes me worried, that the 'Updates tab' states:

When there are security updates: Download and install automatically

If that statement is true, updates will be installed, when some security updates are found, so I may have to wait for more than one night ...

(See the attached screenshot)

sudodus (nio-wiklund) wrote :

After the first night, the used space in 'aufs', the root, had increased from 18M to 40M. I'm not sure if it is caused by a security upgrade or only growing log files, and I don't want to touch the system yet, I will let it run for at least one more night.

Let us say that it was doing an unattended security upgrade last night, but did not bring all the other upgrade candidates along. This will keep the the usage of the available space down to a minimum, and might be a good solution, but it is different from what I was thinking of originally: no upgrades at all for live and persistent live systems.

Brian Murray (brian-murray) wrote :

Comment #26 is due to a separate issue with software-properties and how it naively determines if security updates will be installed automatically. Here's the code in question:

143 def get_update_automation_level(self):
144 """ Parse the apt cron configuration. Try to fit a predefined use case
145 and return it. Special case: if the user made a custom
146 configurtation, that we cannot represent it will return None """
147 if apt_pkg.config.find_i(softwareproperties.CONF_MAP["autoupdate"]) > 0:
148 # Autodownload
149 if apt_pkg.config.find_i(softwareproperties.CONF_MAP["unattended"]) == 1\
150 and os.path.exists("/usr/bin/unattended-upgrade"):
151 return softwareproperties.UPDATE_INST_SEC
152 elif apt_pkg.config.find_i(softwareproperties.CONF_MAP["autodownload"]) == 1 and \
153 apt_pkg.config.find_i(softwareproperties.CONF_MAP["unattended"]) == 0:
154 return softwareproperties.UPDATE_DOWNLOAD
155 elif apt_pkg.config.find_i(softwareproperties.CONF_MAP["unattended"]) == 0 and \
156 apt_pkg.config.find_i(softwareproperties.CONF_MAP["autodownload"]) == 0:
157 return softwareproperties.UPDATE_NOTIFY
158 else:
159 return None

The test is line 149 and 150 assumes that -security is enabled in Allowed-Origins, which I disabled with the casper upload.

Brian Murray (brian-murray) wrote :

With regards to comment #27 this is probably because package list updating is still enabled, /etc/apt/apt.conf.d/10periodic, and I think that's fine. You could look for file dates in /var/lib/apt/lists/ to confirm my suspicion.

description: updated
description: updated
sudodus (nio-wiklund) wrote :

@ bryan-murray,

I looked for file dates in /var/lib/apt/lists/ to confirm your suspicion, and I think you are right. See the attached file.

There was no increase in drive space used after the second night compared to after the first night, so I am convinced that the bug-fix works :-)

Hello sudodus, or anyone else affected,

Accepted casper into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/casper/1.376.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in casper (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed
ventrical (dale-f-beaudoin) wrote :

My question would be: Will this fix be in the xenial/daily/current amd64.iso if I zsync the image ?

Regards

sudodus (nio-wiklund) wrote :

Maybe you need casper 1.379, which is avaiable in Yakkety, while casper 1.376.1 in Xenial is a different version, which might or might not work correctly.

I think casper 1.376.1 is available via xenial-proposed. If you get it via the daily iso file, fine :-)

sudodus (nio-wiklund) wrote :

Alongside your test, ventrical, I started a test using the Ubuntu amd64 current daily iso file (dated 20161009). I created a *persistent live* drive with mkusb (and selected not to use the mkusb tweak to stop unattended upgrades). I'm looking forward to the result tomorrow, if there are any unattended upgrades.

ventrical (dale-f-beaudoin) wrote :

@jbicha,

Thank you for your reply. This makes it a whole lot easier to deploy.

regards..

ventrical (dale-f-beaudoin) wrote :

@nio

is that yakkety or xenial you are testing overnight?

sudodus (nio-wiklund) wrote :

I tested yakkety earlier. Now I am testing xenial.

sudodus (nio-wiklund) wrote :

The fix in xenial does not work. This morning the used space in casper-rw had increased from 66 MiB to 449 MiB (unattended upgrade). See the attached file (in this comment and the next comment),

xenial-pers-live-unattended-update-1.png
xenial-pers-live-unattended-update-2.png

sudodus (nio-wiklund) wrote :
sudodus (nio-wiklund) on 2016-10-11
tags: added: verification-failed
removed: verification-needed
sudodus (nio-wiklund) wrote :

I think we have to check Yakkety again, just to make sure it does not 'get tempted' to install security updates, when they arrive during the night.

If it fails, there is no time to fix it (before Yakkety is released), but at least, there can be a release note about it, also describing how to work around it. And we can be aware of the problem at the Ubuntu Forums.

sudodus (nio-wiklund) wrote :

I started a test using the Ubuntu amd64 Yakkety current daily iso file alias release candidate (dated 20161008). I created a *persistent live* drive with mkusb (and selected not to use the mkusb tweak to stop unattended upgrades). I'm looking forward to the result tomorrow, if there are any unattended upgrades.

sudodus (nio-wiklund) wrote :

I started a similar test using the Ubuntu amd64 Yakkety, an earlier daily iso file that I tried before (dated 20161003). It is also a persistent live system running in another computer.

Brian Murray (brian-murray) wrote :

The file 53disable_unattended_upgrades, located in /usr/share/initramfs-tools/scripts/casper-bottom is not executable so presumably it didn't get run and the -security pocket is still enabled in /etc/apt/apt.conf.d/50unattended_upgrades.

sudodus (nio-wiklund) wrote :

What I did (via mkusb) is described in posts #4 and #5, and it works, when I test it. That will also change the visual output of the 'Updates' tab in 'Software & Updates'. (Seen in the attachment to post #41).

sudodus (nio-wiklund) wrote :

The tests of Ubuntu Yakkety current daily and the previous daily (dated 20161003) have similar results after the night. The biggest files in the casper-rw partition are apt cache files, as described in post #29. But there are also some pyc (python cache?) files and I found even an executable

/media/ubuntu/casper-rw/upper/usr/sbin/anacron.distrib

but I think this is just keeping the system up to date without installing updated versions of program packages from the repositories. See the attached file from the test based on the previous daily (dated 20161003), which would have 'more incentive' to do unattended upgrades, because more packages have changed is the iso file was created.

So it seems to me that the bug is fixed for Yakkety but not for Xenial. Do you think so too, Brian?

sudodus (nio-wiklund) wrote :

Correction: because more packages have changed is the iso file was created
--> because more packages have changed after the iso file was created

Jeremy Bicha (jbicha) wrote :

There are no security updates for yakkety yet since it's not been released. Note comment #45.

Changed in casper (Ubuntu Yakkety):
status: Fix Released → Triaged
Jeremy Bicha (jbicha) wrote :

Re-opening the yakkety task then.

sudodus (nio-wiklund) wrote :

Learning something new again :-)

ventrical (dale-f-beaudoin) wrote :

@nio

I used xenial. I tried to leave report at U+1 but forums are down ?

here is what I pasted:

2016-10-12 03:31:14,661 INFO Initial blacklisted packages:
2016-10-12 03:31:14,661 INFO Initial whitelisted packages:
2016-10-12 03:31:14,661 INFO Starting unattended upgrades script
2016-10-12 03:31:14,662 INFO Allowed origins are: ['o=Ubuntu,a=xenial-security']
2016-10-12 03:31:18,460 INFO No packages found that can be upgraded unattended and no pending auto-removals

so the fix worked

there is a big diff between the previous unattended-upgrades.log and this one. So I am assuming the fix worked.

Jeremy Bicha (jbicha) wrote :

Re-closing the yakkety task since this does work on yakkety.

For those trying to test on xenial, you'll need to wait for casper 1.376.2, still in the unapproved queue. xenial doesn't have security updates every day so testing the latest daily image often will not trigger this bug overnight.

Changed in casper (Ubuntu Yakkety):
status: Triaged → Fix Released

On Wed, Oct 12, 2016 at 06:20:10AM -0000, sudodus wrote:
> The tests of Ubuntu Yakkety current daily and the previous daily (dated
> 20161003) have similar results after the night. The biggest files in the
> casper-rw partition are apt cache files, as described in post #29. But
> there are also some pyc (python cache?) files and I found even an
> executable
>
> /media/ubuntu/casper-rw/upper/usr/sbin/anacron.distrib
>
> but I think this is just keeping the system up to date without
> installing updated versions of program packages from the repositories.
> See the attached file from the test based on the previous daily (dated
> 20161003), which would have 'more incentive' to do unattended upgrades,
> because more packages have changed is the iso file was created.
>
> So it seems to me that the bug is fixed for Yakkety but not for Xenial.
> Do you think so too, Brian?

That's correct this is fixed in Yakkety but not yet in Xenial. I
confirmed this today by booting a Yakkety Live CD and inspecting
/etc/apt/apt.conf.d/50unattended-upgrades and observed that the security
pocket is disabled. Additionally, I checked
/usr/share/initramfs-tools/scripts/casper-bottom/53disable_unattended_upgrades,
the code which changes 50unattended-upgrades and it was executable.

--
Brian Murray

Brian Murray (brian-murray) wrote :

On Wed, Oct 12, 2016 at 02:20:04PM -0000, ventrical wrote:
> @nio
>
> I used xenial. I tried to leave report at U+1 but forums are down ?
>
> here is what I pasted:
>
> 2016-10-12 03:31:14,661 INFO Initial blacklisted packages:
> 2016-10-12 03:31:14,661 INFO Initial whitelisted packages:
> 2016-10-12 03:31:14,661 INFO Starting unattended upgrades script
> 2016-10-12 03:31:14,662 INFO Allowed origins are: ['o=Ubuntu,a=xenial-security']
> 2016-10-12 03:31:18,460 INFO No packages found that can be upgraded unattended and no pending auto-removals
>
> so the fix worked

You shouldn't see "xenial-security", or anything for that matter, in the
log line for Allowed origins if the fix worked.

I've uploaded a new version of casper to the Xenial queue for review by
the SRU team.

--
Brian Murray

ventrical (dale-f-beaudoin) wrote :

@brian-murray

Ok.. thanks for that clarification.

Hello sudodus, or anyone else affected,

Accepted casper into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/casper/1.376.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: removed: verification-failed
tags: added: verification-needed
sudodus (nio-wiklund) wrote :

I started a test using the Ubuntu Xenial amd64 current daily iso file (dated 20161013). I created a *persistent live* drive with mkusb (and selected not to use the mkusb tweak to stop unattended upgrades). The new casper 1.376.2 was in the -proposed repo but not in the iso file, so I updated & upgraded it. (I installed synaptic too, to get a good overview of the repositories etc, so there is more data already in the casper-rw partition.)

I'm looking forward to the result tomorrow and maybe day after tomorrow, if there are any unattended upgrades, and what the log file looks like.

sudodus (nio-wiklund) wrote :

I'm afraid, that we need another bug-fix. There was not much to upgrade, when the test system 'tried' to upgrade, and left the following 'unattended-upgrades.log'
---
2016-10-14 15:43:22,573 INFO Initial blacklisted packages:
2016-10-14 15:43:22,573 INFO Initial whitelisted packages:
2016-10-14 15:43:22,573 INFO Starting unattended upgrades script
2016-10-14 15:43:22,573 INFO Allowed origins are: ['o=Ubuntu,a=xenial-security']
2016-10-14 15:43:24,622 INFO No packages found that can be upgraded unattended and no pending auto-removals
---
See also the attached file (screenshot).

Allowed origins are: ['o=Ubuntu,a=xenial-security'] does not look right.

Brian Murray (brian-murray) wrote :

What are the contents of /etc/apt/apt.conf.d/50unattended-upgrades?

sudodus (nio-wiklund) wrote :

Most of it is commented away, but this is active:
---
Unattended-Upgrade::Allowed-Origins {
 "${distro_id}:${distro_codename}-security";
};
---

Brian Murray (brian-murray) wrote :

I booted a Xenial Live CD, dated 20161014, and the security pocket was disabled in 50unattended-upgrades furthermore running unattended-upgrades -v did not show "xenial-security".

Brian Murray (brian-murray) wrote :

I installed synaptic and launched it and the settings did not change.

I launched and fiddled with software-properties-gtk and the setting did not change.

I guess that just leaves mkusb.

sudodus (nio-wiklund) wrote :

mkusb can modify the apt settings for persistent live drives, but that is to avoid unattended upgrades, and it changes the visual status seen at the 'Updates' tab in 'Software & Updates'. Otherwise mkusb clones the iso file to a partition and extracts some boot stuff (for grub booting) to [another] FAT partition. And for this test, I did *not* let mkusb tamper with any of the files in /etc/apt/apt.conf.d, so I declined the offer in the pop up window, where it can be selected.

-o-

When I made the fix in mkusb for this problem, I inspected the files in the directory

/etc/apt/apt.conf.d

with the settings 'when there are security updates:' for

- Download and install automatically (which causes unattended upgrades) and
- Display automatically (which does not cause unattended upgrades).

After that I simply made mkusb change the files to be exactly like the were for 'Display immediately'. I let mkusb write to corresponding files in the casper-rw partition "$targ1"

And it works when I test it in Xenial and Yakkety. There is no need for this fix in the previous versions.

from the mkusb shell-script:
---
# change: 'Download and install automatically' to 'Display immediately'

if $chk_upgrd
then
 /bin/echo -e \
 "$inversvid set security upgrade action to 'Display immediately' $resetvid"
 umount "$targ1" 2>&1
 mount "${tu}5" "$targ1" 2>&1
 mkdir -p "$targ1"/upper/etc/apt/apt.conf.d/
 echo 'APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
APT::Periodic::Unattended-Upgrade "0";' \
 > "$targ1"/upper/etc/apt/apt.conf.d/10periodic
 cp -p "$targ1"/upper/etc/apt/apt.conf.d/10periodic \
       "$targ1"/upper/etc/apt/apt.conf.d/20auto-upgrades
else
 /bin/echo -e "$inversvid keep current security upgrade settings $resetvid"
fi
---

sudodus (nio-wiklund) wrote :

You can check with a live-only session booted from a pendrive (or DVD disk) cloned using usb-creator-gtk, dd, or mkusb. The file

/etc/apt/apt.conf.d/50unattended-upgrades

is exactly like the one I uploaded from my persistent live session. But it is not used, when 'Display immediately' is selected. So it seems governed by the two files

10periodic
20auto-upgrades

Don't ask me why it is configured like that :-)

sudodus (nio-wiklund) wrote :

The test system has been running for 3 days and nights now, and there has been no unattended upgrade. So either no security upgrade have been uploaded, or the bug-fix works, even with the current settings of the file

/etc/apt/apt.conf.d/50unattended-upgrades

Anyway, you can see the attached screenshot.

-o-

How can we find out, when the latest security upgrade was uploaded for Ubuntu 16.04.1 LTS?

Brian Murray (brian-murray) wrote :

You could look at the xenial-changes mailing list and look for messages with the subject xenial-security.

https://lists.ubuntu.com/archives/xenial-changes/2016-October/date.html

sudodus (nio-wiklund) wrote :

Oct 14:
[ubuntu/xenial-security] tzdata 2016g-0ubuntu0.16.04 (Accepted) Adam Conrad
[ubuntu/xenial-security] openjpeg2 2.1.0-2.1ubuntu0.1 (Accepted) Marc Deslauriers
Oct 17:
[ubuntu/xenial-security] ffmpeg 7:2.8.8-0ubuntu0.16.04.1 (Accepted) Marc Deslauriers

tzdata - up to date (was it updated automatically?)
openjpeg2 - not found (the 'best match' is openjpeg-tools)
ffmpeg - not installed (can I install the old package and check if it will be updated automatically?)

---
ubuntu@ubuntu:~$ apt-cache policy tzdata
tzdata:
  Installed: 2016g-0ubuntu0.16.04
  Candidate: 2016g-0ubuntu0.16.04
  Version table:
 *** 2016g-0ubuntu0.16.04 500
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     2016d-0ubuntu0.16.04 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
ubuntu@ubuntu:~$ apt-cache policy openjpeg2
N: Unable to locate package openjpeg2
ubuntu@ubuntu:~$ apt-cache policy ffmpeg
ffmpeg:
  Installed: (none)
  Candidate: 7:2.8.6-1ubuntu2
  Version table:
     7:2.8.6-1ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
ubuntu@ubuntu:~$ apt-cache policy openjpeg
N: Unable to locate package openjpeg
ubuntu@ubuntu:~$ apt-cache policy open
Display all 306 possibilities? (y or n)
ubuntu@ubuntu:~$ apt-cache policy openj
openjade openjdk-8-jre-jamvm openjdk-9-source
openjade1.3 openjdk-8-jre-zero openjfx
openjdk-8-dbg openjdk-8-source openjfx-source
openjdk-8-demo openjdk-9-dbg openjpeg-tools
openjdk-8-doc openjdk-9-demo openjpip-dec-server
openjdk-8-jdk openjdk-9-doc openjpip-server
openjdk-8-jdk-headless openjdk-9-jdk openjpip-viewer
openjdk-8-jre openjdk-9-jdk-headless openjpip-viewer-xerces
openjdk-8-jre-dcevm openjdk-9-jre
openjdk-8-jre-headless openjdk-9-jre-headless
ubuntu@ubuntu:~$ apt-cache policy openjpeg-tools
openjpeg-tools:
  Installed: (none)
  Candidate: 1:1.5.2-3.1
  Version table:
     1:1.5.2-3.1 500
        500 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages

ubuntu@ubuntu:~$
---

sudodus (nio-wiklund) wrote :

I started another persistent live session from the released Ubuntu 16.04.1 LTS amd64 iso file. There are several security updates, for example for tzdata (f-->g)

tzdata:
  Installed: 2016f-0ubuntu0.16.04
  Candidate: 2016g-0ubuntu0.16.04

I have upgraded casper and I will expose this system for the temptation to upgrade unattended.

sudodus (nio-wiklund) wrote :

I am very sorry, but the persistent live system I started yesterday fell for the temptation to upgrade unattemded. The used space increased from 259 MiB to 1,3 GiB. See the attached screenshot.

-o-

Are there other things, that have changed between the released 16.04.1 LTS and the current daily iso file (except casper and the number of packages classified as security updates), that could make a difference regarding unattended upgrades?

-o-

Maybe you should try along the path I used in mkusb - to change the files

10periodic
20auto-upgrades

and restore all the other fixes. I hope and think it would be possible from casper too.

Brian Murray (brian-murray) wrote :

I created an Ubuntu USB disk using usb-creator-gtk using the Xenial daily iso from 20161025. I booted the usb disk and discovered that all the settings are as they should be and running unattended-upgrades does nothing. I'm marking this as verification done as mkusb is not a tool which is included in the Ubuntu archives.

Attached is a picture of my laptop running the Live CD with all the verification done.

tags: added: verification-done
removed: verification-needed
sudodus (nio-wiklund) wrote :

Even if it works for you with a live seesion, it does not work for me with a persistent live session using a casper-rw partition. starting from the Ubuntu 16.04.1 LTS iso file. I don't think mkusb has anything at all to to with this functionality because it does not touch the mechanism (unless you tell it to do it, and then it will prevent unattended upgrades).

mkusb has a work-around feature, and I think all other tools and manual methods, that create persistent live drives will need some work-around to avoid this problem. For example, if you create a cloned system with the Ubuntu Startup Disk Creator, the cloned result will be exactly like cloning with mkusb. It is possible in such cases to create persistence by using a casper-rw partition (or file) in another drive. I will test tonight, if such a system is also vulnerable to this bug.

I plan to start the test from the Ubuntu 16.04.1 LTS desktop amd64 iso file. I intend to upgrade casper, but no other program package unless you advice what must also be upgraded for your fix to work.

But don't tell me that I must try with the current daily iso file, because then I would have to test for a very long period of time for something to happen. In that case (if you know things have changed to the better, but don't know which program packages are involved) maybe I should zsync today's daily iso file and keep it for some weeks before I start it and let it run for at least 24 hours.

The important thing here is to get something working for the next point release, 16.04.2 LTS, so I think it should be tested thoroughly.

Download full text (6.0 KiB)

I will be watching for your results, I have not been able to get persistent
partitions working with 64bit 16.04 using UNetbootin or SDC.
Drive fails to boot similar to having a full casper-rw.

On Wed, Oct 26, 2016 at 10:16 AM, sudodus <email address hidden>
wrote:

> Even if it works for you with a live seesion, it does not work for me
> with a persistent live session using a casper-rw partition. starting
> from the Ubuntu 16.04.1 LTS iso file. I don't think mkusb has anything
> at all to to with this functionality because it does not touch the
> mechanism (unless you tell it to do it, and then it will prevent
> unattended upgrades).
>
> mkusb has a work-around feature, and I think all other tools and manual
> methods, that create persistent live drives will need some work-around
> to avoid this problem. For example, if you create a cloned system with
> the Ubuntu Startup Disk Creator, the cloned result will be exactly like
> cloning with mkusb. It is possible in such cases to create persistence
> by using a casper-rw partition (or file) in another drive. I will test
> tonight, if such a system is also vulnerable to this bug.
>
> I plan to start the test from the Ubuntu 16.04.1 LTS desktop amd64 iso
> file. I intend to upgrade casper, but no other program package unless
> you advice what must also be upgraded for your fix to work.
>
> But don't tell me that I must try with the current daily iso file,
> because then I would have to test for a very long period of time for
> something to happen. In that case (if you know things have changed to
> the better, but don't know which program packages are involved) maybe I
> should zsync today's daily iso file and keep it for some weeks before I
> start it and let it run for at least 24 hours.
>
> The important thing here is to get something working for the next point
> release, 16.04.2 LTS, so I think it should be tested thoroughly.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1619188
>
> Title:
> Unattended upgrades can break persistent live media
>
> Status in casper package in Ubuntu:
> Fix Released
> Status in casper source package in Xenial:
> Fix Committed
> Status in casper source package in Yakkety:
> Fix Released
>
> Bug description:
> Test Case
> ---------
> 1) Boot an Ubuntu 16.04 Live CD
> 2) Choose Try Ubuntu
> 3) head -n5 /etc/apt/apt.conf.d/50unattended-upgrades
> 4) Observe -security is enabled in line 3. (// is a comment)
> 5) Run /usr/lib/apt/apt.systemd.daily
> 6) Observe your Live environment run out of space! (I received a pop-up
> re lack of free space and /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
> contained an error installing a package due to disk full.)
>
> With the version of the package in -proposed step 4 will reveal the
> -security pocket being disabled and step 6 won't fill your live
> environment.
>
> Regression Potential
> --------------------
> Persistent live users will not receive updates from -security, but that
> seems less bad than destroying people's live environment by filling up
> their disk.
>
> Original Bug Descriptio...

Read more...

sudodus (nio-wiklund) wrote :
Download full text (3.2 KiB)

The tests started yesterday can be described like this:

1. I tested the Ubuntu 16.04.1 LTS desktop amd64 iso file.

2. To be completely sure, I created two USB boot drives with the Ubuntu Startup Disk Creator (not that it would clone differently from mkusb, but anyway, just in case.

3. I upgraded casper to the new version in xenial-proposed, 1.376.2.

4. I ran both tests in UEFI mode, a live-only session in an Intel NUC

https://www-ssl.intel.com/content/www/us/en/nuc/nuc-kit-nuc6i3syh.html

and a persistent live session in a Lenovo X131e

https://shop.lenovo.com/ISS_Static/ww/wci/products/us/laptop/thinkpad/x-series/x131e-intel/X131e-Datasheet-Intel.pdf

The persistence was using a casper-rw partition in another drive (because the cloned drive has the read-only file system ISO 9660.

I checked with 'df -h' how much space was used before and after the test period.

5. Result

Live-only:

ubuntu@ubuntu:~$ date;df -h .
Wed Oct 26 17:46:20 UTC 2016
Filesystem Size Used Avail Use% Mounted on
/cow 1.9G 48M 1.9G 3% /
ubuntu@ubuntu:~$ echo 'this is live-only'
this is live-only
ubuntu@ubuntu:~$ echo 'this is live-only'
this is live-only
ubuntu@ubuntu:~$ date;df -h .
Wed Oct 26 19:38:35 UTC 2016
Filesystem Size Used Avail Use% Mounted on
/cow 1.9G 48M 1.9G 3% /
ubuntu@ubuntu:~$ date;df -h .
Thu Oct 27 16:30:19 UTC 2016
Filesystem Size Used Avail Use% Mounted on
/cow 1.9G 1.1G 886M 54% /
ubuntu@ubuntu:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
tmpfs 384M 6.4M 378M 2% /run
/dev/sdb 1.5G 1.5G 0 100% /cdrom
/dev/loop0 1.4G 1.4G 0 100% /rofs
/cow 1.9G 1.1G 886M 54% /
tmpfs 1.9G 172K 1.9G 1% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
tmpfs 1.9G 4.0K 1.9G 1% /tmp
tmpfs 384M 124K 384M 1% /run/user/999
/dev/mmcblk0p2 6.0M 0 6.0M 0% /media/ubuntu/Firmware

Persistent live:

ubuntu@ubuntu:~$ date;df -h .
Wed Oct 26 18:39:29 UTC 2016
Filesystem Size Used Avail Use% Mounted on
/cow 22G 92M 21G 1% /
ubuntu@ubuntu:~$ date;df -h .
Thu Oct 27 16:29:50 UTC 2016
Filesystem Size Used Avail Use% Mounted on
/cow 22G 1.1G 20G 6% /
ubuntu@ubuntu:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
tmpfs 380M 6.2M 374M 2% /run
/dev/sda 1.5G 1.5G 0 100% /cdrom
/dev/loop0 1.4G 1.4G 0 100% /rofs
/cow 22G 1.1G 20G 6% /
tmpfs 1.9G 228K 1.9G 1% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
tmpfs 1.9G 4.0K 1.9G 1% /tmp
tmpfs 380M 92K 380M 1% /run/user/999
/dev/mmcblk0p5 22G 1.1G 20G 6% /media/ubuntu/casper-rw
/dev/mmcblk0p3 6.4G 2.1G 4.0G 34% /media/ubuntu/root

6. Conclusion

In both cases, a big unattended upgrade was performed. The used space increased

from 48M to 1.1G in the live-only case and
from 92M to 1.1G in the persistent liv...

Read more...

sudodus (nio-wiklund) wrote :
Brian Murray (brian-murray) wrote :

Oh, I think I get it now. casper makes changes at boot so updating to the new version from an already booted image won't do anything.

sudodus (nio-wiklund) wrote :

So it is not possible to use your fix in a live-only session.

What about a persistent live session? Will casper do these things before the overlay structure is read or after? In other words, is is necessary that the fix is included in the iso file to work?

Brian Murray (brian-murray) wrote :

On Thu, Oct 27, 2016 at 08:00:42PM -0000, sudodus wrote:
> So it is not possible to use your fix in a live-only session.

It is possible to use the fix in a live-only session, you just need to
use a daily build or wait for the 16.04.2 iso.

> What about a persistent live session? Will casper do these things before
> the overlay structure is read or after? In other words, is is necessary
> that the fix is included in the iso file to work?

The updated version of casper needs to be included in the iso file.

--
Brian Murray

sudodus (nio-wiklund) wrote :

Too bad that we were not told at once, that we must start with an iso file that already has your fix. The tip to add xenial-proposed and upgrade caused a lot of FUD and extra work.

Comments 32,57:
---
Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.
---

Anyway, now we think that we know.

Because of the FUD I will wait some week(s) to 'collect' security updates, and try with the iso file from a couple of days ago. I saved it as

$ ls -l xenial-desktop-amd64-2016-10-26.iso
... 1525235712 okt 26 07:07 xenial-desktop-amd64-2016-10-26.iso
$ md5sum xenial-desktop-amd64-2016-10-26.iso
287daf005fb1351b4648aed19434f82f xenial-desktop-amd64-2016-10-26.iso

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package casper - 1.376.2

---------------
casper (1.376.2) xenial-proposed; urgency=medium

  * Make scripts/casper-bottom/53disable_unattended_upgrades executable.
    (LP: #1619188)

 -- Brian Murray <email address hidden> Tue, 11 Oct 2016 09:32:27 -0700

Changed in casper (Ubuntu Xenial):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for casper has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

sudodus (nio-wiklund) wrote :

I am running a long time test now (since approximately 24 hours). There are fresh security updates (for example of python), that appeared after October 26 when the daily iso file was saved. It is promising; I think the bug-fix works, but I will let it run for a second night before I finish this test.

sudodus (nio-wiklund) wrote :

I am finishing the long time test now after the second night. There were no unattended upgrades in this Ubuntu live system. I can confirm that the bug-fix works in a live-only system :-)

sudodus (nio-wiklund) wrote :

I am finishing the test with a persistent live system made with mkusb after two nights.

I did not let mkusb change 'what to do, when security updates are detected', so the corresponding system from the 16.04.1 LTS iso file would perform unattended upgrades. The used drive space increased from 66 MiB to 88 MiB. I did not check which files were written, but I assume some log files. This is far from the big increase of used space, when the system performs unattended upgrades.

So I can confirm that the bugfix in casper works also in a persistent live system. We can look forward to the next point release, Ubuntu 16.04.2 LTS, where it will be implemented. Today it is possible to grab the Ubuntu Xenial daily iso file and make a live drive without unattended upgrades :-)

To post a comment you must log in.