OTA and u-d-f fail if image is too big for the system partition

[Lukyk]

On Meizu MX4 upgrade from OTA-9.1 to OTA-10 fails.
Upgrade downloads normally, phone restarts and starts system upgrade and after a while it freezes with strange icon (small microchip with red sign on it)

When you turn off the phone and start it again it still presents itself as OTA-9.1 and again it founds and offer upgrade.
Anyway some changes from OTA-10 seems to be applied, because I can see design changes, copy function in web browser etc.

It's reproducible on my phone any time I try to upgrade. If I can help with output of some log, or anything else, please let me know.

Well described here by Lars Kristian Wichmann Hansen:
http://askubuntu.com/questions/754963/upgrade-meizu-mx4-ubuntu-edition-to-ota-10-fails

* u-d-f case described in bug 1582325

[Egbert Verhage (eggiecode)]

I got the same problem with my BQ Aquaris E5, with version 11.

Now with version 12 is looks like the phone is updating, but after it is done is still says that it is on OTA-9.1

[Pat McGowan (pat-mcgowan)]

try updating from command line via
sudo system-image-cli -v

[Ken VanDine (ken-vandine)]

This might be a bug in ubuntu-system-image, but more likely something in applying the update itself in recovery. Not sure where to assign this bug, but it sounds critical to me.

[Eph Zero (eph-zero-j)]

I'm having the exact same experience with a Nexus 7 (including the chip icon). It does seem that all or most of the OTA-10 features have been updated, but the system still thinks it's 9.1 and keeps offering to upgrade.

[Eph Zero (eph-zero-j)]

New information: I just tried again, this time with a writable image, and issuing via ADB:

ubuntu-device-flash touch --channel=ubuntu-touch/stable/ubuntu

That worked!

[Lukyk (luk-wimm)]

Good news, that it's doable by ADB.

Anyway here is an output of sudo system-image-cli -v:

phablet@ubuntu-phablet:~$ sudo system-image-cli -v
[sudo] password for phablet:
[systemimage] Apr 13 09:42:37 2016 (4190) running state machine
[ubuntu-touch/stable/meizu.en/arale]
[systemimage] Apr 13 09:42:37 2016 (4190) Looking for blacklist:
https://system-image.ubuntu.com/gpg/blacklist.tar.xz
[systemimage] Apr 13 09:42:37 2016 (4190) [0xb5826290] Requesting group
download:
       https://system-image.ubuntu.com/gpg/blacklist.tar.xz ->
/var/lib/system-image/keyring.tar.xz
       https://system-image.ubuntu.com/gpg/blacklist.tar.xz.asc ->
/var/lib/system-image/keyring.tar.xz.asc

[systemimage] Apr 13 09:42:37 2016 (4190) Allow GSM? No
[systemimage] Apr 13 09:42:37 2016 (4190)
[/com/canonical/applications/download/com_2eubuntu_2eterminal_5fterminal_5f0_2e7_2e190/883872ab0c7f4425b4764db9eda977a2]
Running group download reactor
[systemimage] Apr 13 09:42:37 2016 (4190) self: <UDMDownloadManager at
0xb5826290>, self._iface: <Interface <ProxyObject wrapping
<dbus._dbus.SystemBus (system) at 0xb604c2d0> :1.135
/com/canonical/applications/download/com_2eubuntu_2eterminal_5fterminal_5f0_2e7_2e190/883872ab0c7f4425b4764db9eda977a2
at 0xb5826530> implementing 'com.canonical.applications.GroupDownload' at
0xb58262f0>
[systemimage] Apr 13 09:42:37 2016 (4190)
[/com/canonical/applications/download/com_2eubuntu_2eterminal_5fterminal_5f0_2e7_2e190/883872ab0c7f4425b4764db9eda977a2]
Group download reactor done
[systemimage] Apr 13 09:42:37 2016 (4190) uncaught exception in state
machine
Traceback (most recent call last):
 File "/usr/lib/python3/dist-packages/systemimage/state.py", line 133, in
__next__
   step()
 File "/usr/lib/python3/dist-packages/systemimage/state.py", line 219, in
_get_blacklist_1
   get_keyring('blacklist', url, 'image-master')
 File "/usr/lib/python3/dist-packages/systemimage/keyring.py", line 107,
in get_keyring
   (ascxz_src, ascxz_dst),
 File "/usr/lib/python3/dist-packages/systemimage/download.py", line 209,
in get_files
   self._get_files(records, pausable, signal_started)
 File "/usr/lib/python3/dist-packages/systemimage/udm.py", line 189, in
_get_files
   missing, local_paths))
AssertionError: Missing destination files:
['/var/lib/system-image/keyring.tar.xz',
'/var/lib/system-image/keyring.tar.xz.asc']
local_paths: [dbus.String('/tmp/blacklist (2).tar.xz'),
dbus.String('/tmp/blacklist (2).tar.xz.asc')]
Exception occurred during update; see log file for details
[systemimage] Apr 13 09:42:37 2016 (4190) system-image-cli exception
Traceback (most recent call last):
 File "/usr/lib/python3/dist-packages/systemimage/main.py", line 402, in
main
   list(state)
 File "/usr/lib/python3/dist-packages/systemimage/state.py", line 133, in
__next__
   step()
 File "/usr/lib/python3/dist-packages/systemimage/state.py", line 219, in
_get_blacklist_1
   get_keyring('blacklist', url, 'image-master')
 File "/usr/lib/python3/dist-packages/systemimage/keyring.py", line 107,
in get_keyring
   (ascxz_src, ascxz_dst),
 File "/usr/lib/python3/dist-packages/systemimage/download.py",...

[Barry Warsaw (barry)]

This looks like an old udm bug coming back, although I can't find the bug number now.

si calls udm telling it both the source (url) and destination (local fs) of the files it wants to download. The old udm bug used to occasionally and unpredictably lead to a situation where udm claims the files have been successfully downloaded (via D-Bus signal), but then when si goes to find the files, they don't exist in the requested destination location. For that reason, I added the assertion to si so we could at least catch this problem when it occurs.

I don't remember the resolution in udm; it either just went away or was fixed at one point. But it looks like the bug is back, so I've added a udm bug task.

One way to double check this is to try the following in the shell:

$ SYSTEMIMAGE_PYCURL=1 system-image-cli -vv

This tells si to bypass udm and use the built-in pycurl based downloader. You won't get the traceback (since it originates in the udm.py module and that won't be used), but if you do have any other problem with downloading then it may indeed been an si bug. But I suspect whatever udm problem we had before is back again.

[Michael Sheldon (michael-sheldon)]

Barry - If you're unable to find the previous UDM bug report (I had a bit of a search, but couldn't find it either), could you comment with any extra details you can remember from it? Since it pre-dates me working on UDM I don't really have any knowledge of it.

In my search I also came across this bug https://bugs.launchpad.net/ubuntu-download-manager/+bug/1279965 which curiously was marked as invalid without explanation (although I'd have thought with the hash checking, attempts at updates with corrupted downloads shouldn't be possible any more), might be worth double checking though.

[Lukyk (luk-wimm)]

The command ou provided:
$ sudo SYSTEMIMAGE_PYCURL=1 system-image-cli -vv
seems to download some files, then continued with restart -> update and ended with the same fail (with chip icon) as normally.

Everything was very quick, if you want me to post some log output, please point me to its location.

[Barry Warsaw (barry)]

Michael: I do vaguely remember talking with Manuel about that. I don't know why it's been marked invalid either.

@Lukyk: /var/log/system-image/client.log please.

[Ken VanDine (ken-vandine)]

This looks like it's actually failing in applying the update in recovery. Please get us these logs:

/cache/recovery/log
/cache/recovery/last_log

[Barry Warsaw (barry)]

@ken-vandine: If it's failing in recovery, then it's not a system-image problem. Can we mark that bugtask Invalid?

[Lukyk (luk-wimm)]

Here are those requested logs. There was no last_log, but last_log_r instead.

[Barry Warsaw (barry)]

From last_log_r:

e2fsck 1.41.14 (22-Dec-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/block/platform/mtk-msdc.0/by-name/system: 64906/132192 files (0.2% non-contiguous), 523388/528128 blocks
Applying update: ubuntu-1159f4dbbd369d693a371b78c9df7550531ba346b20ade297214ff4c7c7eb7ad.delta-ubuntu-5f4806a2d26af9d26279d1e66d217281dfa19219a85d2fd64999460f47d6efb9.tar.xz
tar: write error: No space left on device
E:Ubuntu update failed
fw_upgrade_finish
Slowly SD retry failed (/dev/block/mmcblk1p1)
W:failed to mount /dev/block/mmcblk1p1 (No such file or directory); trying /dev/block/mmcblk1
E:failed to mount /sdcard (No such file or directory)
E:unknown volume for path [/sdcard2/logs/recovery]
E:unknown volume for path [/sdcard1/logs/recovery]
Ubuntu update complete.
I:Ubuntu update complete.

So it looks like the partition is out of space. I wonder if that could be the cause of the AssertionError in the client, possibly because either udm isn't noticing the ENOSPACE and returning a success code anyway?

Take a look at /android/cache/recovery. Delete any big .xz files (and the associated .asc files) you find there and try again. But maybe there just isn't enough space available if the updates are big enough. In that case, delete those files again and try forcing a full update, e.g. `system-image-cli -b 0 -vv` to see if those are small enough to fit in the available space.

[Jean-Baptiste Lallement (jibel)]

Could you please connect to the system, run the command df and paste the result in this bug report. Thanks.

[Jean-Baptiste Lallement (jibel)]

There is a significant size increase of the rootfs between OTA9 and 10
OTA8 1 128 550 400
OTA9 1 130 895 360
OTA10 1 196 871 680
OTA10.1 1 196 922 880

among the changes anthy (japanese input method) is responsible for a fair amount of this increase (+45MB)

[Jean-Baptiste Lallement (jibel)]

On my MX4 there is only 52MB left on the system partitition

/dev/disk/by-partlabel/system 2046272 1977696 52192 98% /

But on another MX4 running the same version there is 73MB left.

[Jean-Baptiste Lallement (jibel)]

The custom tarballs increased by nearly 100MB between OTA9 and 10
OTA9 193 013 760
OTA10 290 938 880

[Lukyk (luk-wimm)]

You were right, system partition was 100% full. I was checking available space, but didn't noticed, that system has different partition. In my case, this problem is probably my fault, because I've been playing with some manually installed x-applications.

But it's weird, that this problem occured in other cases, for example Lars Kristian Wichmann Hansen stated, that he didn't do any nonstandard operations.

Anyway sorry for causing troubles.
I will probably do manual reflash to get clean install now.

[Barry Warsaw (barry)]

Thanks for the update. Good luck!

[Jean-Baptiste Lallement (jibel)]

@Lukyk, thanks for following up. We've filed bug 1570573 to track the size of the rootfs which is becoming an issue.

@Barry could anything be done to estimate the space required to do an upgrade and stop before proceeding instead of leaving the system in a broken state.

[Barry Warsaw (barry)]

On Apr 15, 2016, at 01:51 PM, Jean-Baptiste Lallement wrote:

>@Barry could anything be done to estimate the space required to do an
>upgrade and stop before proceeding instead of leaving the system in a
>broken state.

We could probably do some checks of the various file systems involved, but
it's difficult to know how much additional space will be consumed by an
update. The images do have a size parameter, but that's the size of the
compressed data files - they don't tell us anything about how much additional
space is required once they're unpacked and installed.

The other question is what to do if there's no space available. While I agree
that the current situation isn't at all great, if we don't have a way for the
user (or even better, the system) to mitigate space issues, then it doesn't
make the situation that much better. The device still won't successfully
update, although we would be able to tell them why. We might be able to solve
that problem more easily though, by having system settings inspect the
recovery log file for errors after a reboot.

[Dario Cavedon (dcavedon)]

Don't know if it's the same problem, I have a MX4 running an old Ubuntu Touch version r1, with the same problem. When I download a new version and try to install it, I always get this error (see image http://imgur.com/XLxG4ql ).
So I try updating it via ubuntu-device-flash

ubuntu-device-flash touch --channel=ubuntu-touch/stable/ubuntu

I get this error

error pushing: failed to copy '/home/dario/.cache/ubuntuimages/pool/ubuntu-b5f4cc2c8fdcda0c8c2f1112f8ea22648af9fe2306e9038147427e8306c2acb2.tar.xz' to '/cache/recovery//ubuntu-b5f4cc2c8fdcda0c8c2f1112f8ea22648af9fe2306e9038147427e8306c2acb2.tar.xz': No space left on device

[Lukyk (luk-wimm)]

@Dario I was succesfull with added --wipe parameter. But be aware, that all data will be lost.

[Lukyk (luk-wimm)]

btw: I also don't recommend the channel you've mentioned. It caused troubles in my case (I was unable to get latest OTA and install some core apps).
I was succesfull with this command (if you have european version of Meizu):
ubuntu-device-flash touch --channel ubuntu-touch/stable/meizu.en --wipe

[Barry Warsaw (barry)]

Just to reiterate; currently system-image-client can't do any pre-calculation to decide whether there is enough space to unpack the update or not, because there is no server information on the unpacked size.

The best we can do with the current design would be for si-client or system-settings to parse the recovery log file after reboot and somehow look for the no space left message.

[Pat McGowan (pat-mcgowan)]

It would be preferable to do an up front check and inform settings so they can pop up a warning.
We could make an assumption on the installed size based on the compressed size, perhaps choose a worst case % so we would verify that a factor larger is available. I can try to come up with a heuristic. (or maybe we can add an API to the server to tell us?)

As this so far only happens on devices where the user explicitly enabled r/w and installed packages I will drop the priority a bit.

[Alan Bell (alanbell)]

well I for one did once enable r/w and installed packages (don't think it was much more than pastebinit) however around OTA9 I reflashed the device destroying data and returning to factory, when through the startup wizard and everything and I have not enable r/w since then, and my phone won't install OTA11. So, either it does happen for people who haven't enabled r/w or the process of reflashing over adb doesn't actually clear out all the stuff it was supposed to do.

[Pat McGowan (pat-mcgowan)]

@Alan I think you are right, trying to understand what is not getting cleaned up
Doing a flash will get the latest but not free the space you should have, around 400MB

[Pat McGowan (pat-mcgowan)]

adding an android task
The error from the system-image-updater script should be used to display an error condition when updates are not applied in recovery

[Pat McGowan (pat-mcgowan)]

@alan I did get 1GB back with a simple flash, wipe and bootstrap did not really improve on that on this device. Lets see the results for df -h when you do. I have 1.7GB installed in that partition.

You could also poke around in places like /var/log to see if there are old files remaining.

[Alan Bell (alanbell)]

I have 27MB of stuff in /var/log
there is a dpkg.log.1 showing stuff installed on 2016-02-17 showing stuff being unpacked and configured, I think that is first boot stuff from when I wiped it, no packages installed after that.
looking at where the space is used

phablet@ubuntu-phablet:/$ sudo du -sh *
315M android
6.7M bin
4.0K boot
0 cache
295M custom
0 data
4.0K dev
8.6M etc
0 factory
0 firmware
501M home
33M lib
16K lost+found
788M media
4.0K mnt
191M opt
0 persist
du: cannot access ‘proc/21112/task/21112/fd/4’: No such file or directory
du: cannot access ‘proc/21112/task/21112/fdinfo/4’: No such file or directory
du: cannot access ‘proc/21112/fd/4’: No such file or directory
du: cannot access ‘proc/21112/fdinfo/4’: No such file or directory
0 proc
12K root
488K run
7.2M sbin
4.0K srv
0 sys
0 system
28K tmp
72M userdata
1.2G usr
461M var
0 vendor

/usr /var and /custom seem to account for the space used

[Jean-Baptiste Lallement (jibel)]

@Barry and Pat, as I understand the delta update process, it starts by removing files listed in the "remove" list then unpack everything on top of the system partition. This way to proceed has 2 consequences:
1. It can break at any point once there is no space left. So parsing the log afterwards to report that update failed won't help because depending on when tar failed and which files have been updated, the result is undetermined and there is no guarantee that the system will even boot.
2. Calculating the size of the update on the client side will be extremely expensive. For each file in the tarball, the upgrader would have to calculate the size difference between this file and the same file on the filesystem, sum it and compare to the free space in the system partition.

A possibility would be to calculate the size difference server side, during the calculation of the delta, and store this information in the json index for delta type updates. Then system-image can use this information to decide whether or not an upgrade is possible.

[Alan Bell (alanbell)]

after flashing over USB I have this

phablet@ubuntu-phablet:/$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 479M 4.0K 479M 1% /dev
tmpfs 97M 348K 96M 1% /run
/dev/mmcblk0p7 4.4G 840M 3.4G 20% /userdata
/dev/mmcblk0p6 2.0G 1.6G 392M 81% /
/dev/loop0 141M 139M 1.8M 99% /android/system
none 4.0K 0 4.0K 0% /android
tmpfs 481M 4.0K 481M 1% /etc/fstab
/dev/disk/by-path/platform-mtk-msdc.0-part5 688M 12M 677M 2% /android/cache
none 4.0K 0 4.0K 0% /sys/fs/cgroup
tmpfs 481M 2.1M 479M 1% /tmp
cgmfs 100K 0 100K 0% /run/cgmanager/fs
none 5.0M 0 5.0M 0% /run/lock
none 481M 7.6M 473M 2% /run/shm
none 100M 0 100M 0% /run/user
tmpfs 481M 0 481M 0% /media
tmpfs 481M 8.0K 481M 1% /var/lib/sudo
tmpfs 97M 48K 97M 1% /run/user/32011
tmpfs 97M 0 97M 0% /run/user/0
/dev/mmcblk1p1 1.9G 794M 1.1G 42% /media/phablet/9016-4EF8

phablet@ubuntu-phablet:/$ sudo du -sh *
129M android
6.7M bin
4.0K boot
0 cache
206M custom
0 data
4.0K dev
7.3M etc
0 factory
0 firmware
429M home
33M lib
4.0K lost+found
788M media
4.0K mnt
191M opt
0 persist
du: cannot access ‘proc/948/task/1554/fdinfo/74’: No such file or directory
du: cannot access ‘proc/948/task/1554/fdinfo/77’: No such file or directory
du: cannot access ‘proc/20526/task/20526/fd/4’: No such file or directory
du: cannot access ‘proc/20526/task/20526/fdinfo/4’: No such file or directory
du: cannot access ‘proc/20526/fd/4’: No such file or directory
du: cannot access ‘proc/20526/fdinfo/4’: No such file or directory
0 proc
12K root
4.7M run
7.2M sbin
4.0K srv
0 sys
0 system
2.1M tmp
76M userdata
1.1G usr
260M var
0 vendor

[Barry Warsaw (barry)]

On Jun 03, 2016, at 08:49 AM, Jean-Baptiste Lallement wrote:

>A possibility would be to calculate the size difference server side, during
>the calculation of the delta, and store this information in the json index
>for delta type updates. Then system-image can use this information to decide
>whether or not an upgrade is possible.

I think that's the only viable way to proceed.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mihael (mihaelmilea)]

I have a Meizu Pro 5 which was originally purchased with android. I installed Ubuntu Touch from the stable channel. I try now to switch to rc-proposed channed using system-image-cli and I have the error of "no space left on device". That happens because the cache partition (/dev/sda43) is only 490MB when, in Ubuntu Touch Meizu Pro 5, that partition is 700MB. So, the tar.xz files do not fit into the 490MB - I would need 10 more MB. Can anybody help me with a way to increase the cache partition? Nothing works: fdisk, parted, mke2fs - not even adb shell while the phone is booted into recovery.