BQ E4.5 (r24): after fine update to r25 system is reported as r-1 and as "never" updated

[Matthias Apitz]

After the vacation I have updated today (September 13, 2015) my BQ 4.5 and the one of my wife to r25. Both updates went without any problem and mine does not show any issue. The one of my wife shows in About the device window:

OS number: -1 (and not r25)
last updated: never
version: Ubuntu 15.04-armhf (20150825-170536)

What went wrong and what can I do? I'm afraid that the next update will
fail completely now because it can not see the current version.

Apart of this the device is working.

[Matthias Apitz (gubu)]

in /var/log/syslog it is saying all time:

Sep 13 17:17:05 ubuntu-phablet dbus[826]: [system] Activated service 'com.canonical.SystemImage' failed: Launch helper exited with unknown return code 1
Sep 13 17:17:05 ubuntu-phablet dbus[826]: [system] Activating service name='com.canonical.SystemImage' (using servicehelper)
Sep 13 17:17:08 ubuntu-phablet dbus[826]: [system] Activated service 'com.canonical.SystemImage' failed: Launch helper exited with unknown return code 1
Sep 13 17:17:08 ubuntu-phablet dbus[826]: [system] Activating service name='com.canonical.SystemImage' (using servicehelper)
Sep 13 17:17:11 ubuntu-phablet dbus[826]: [system] Activated service 'com.canonical.SystemImage' failed: Launch helper exited with unknown return code 1
Sep 13 17:17:12 ubuntu-phablet dbus[826]: [system] Activating service name='com.canonical.SystemImage' (using servicehelper)

I will attach as well two screens from the 'About the device' window:

[Matthias Apitz (gubu)]

In addition to the described problems: it is not possible to administer the scopes anymore; one can pull the menu up from the bottom, but can not do any change there.

[Pat McGowan (pat-mcgowan)]

per Matthias

I tried to understand what the py scripts wanna do and it is the
problem that the /etc/system-image/channel.ini file is empty:

$ ls -l /etc/system-image/channel.ini
-rw-r--r-- 1 root root 0 sep 2 11:33 /etc/system-image/channel.ini
$ ls -l /etc/system-image/
total 24
-rw-r--r-- 1 root root 1468 oct 29 2014 archive-master.tar.xz
-rw-r--r-- 1 root root 836 oct 29 2014 archive-master.tar.xz.asc
-rw-r--r-- 1 root root 0 sep 2 11:33 channel.ini
-rw-r--r-- 1 root root 877 oct 27 2014 client.ini
drwxrwxr-x 2 root root 4096 sep 13 08:54 config.d
-rw-r--r-- 1 root root 4662 may 20 18:12 writable-paths

while on my working r25 it has some correct looking values and all other
files have the same size as above;

[Pat McGowan (pat-mcgowan)]

an empty channel.ini file is certainly not right. You can copy the one from the other device you have. It should look something like

 adb shell system-image-cli -i

current build number: 25
device name: krillin
channel: ubuntu-touch/stable/bq-aquaris.en
last update: 2015-09-09 09:13:11
version version: 25
version ubuntu: 20150825.1
version device: 20150821-736d127
version custom: 20150821-887-33-32-vivid

If you want to be sure you actually have all of r25 ota6 you could do this after restoring the channel.ini file

sudo system-image-cli -b 0

This will reinstallll the entire image but not remove any data

[Matthias Apitz (gubu)]

I copied the channel.ini file and now it says:

phablet@ubuntu-phablet:~$ system-image-cli -i
current build number: 25
device name: krillin
channel: ubuntu-touch/stable/bq-aquaris.en
last update: 2015-09-13 08:54:13
version version: 25
version ubuntu: 20150825.1
version device: 20150821-736d127
version custom: 20150821-887-33-32-vivid

but the

sudo system-image-cli -b 0

has no visible effect, it just returns to the shell prompt and even after reboot the scopes are not administrable

[Barry Warsaw (barry)]

On Sep 15, 2015, at 07:22 PM, Matthias Apitz wrote:

>sudo system-image-cli -b 0
>
>has no visible effect, it just returns to the shell prompt and even
>after reboot the scopes are not administrable

Try adding -v or -vv to the command to get console logging. Otherwise, look
at /var/log/system-image/client.log for details.

[Matthias Apitz (gubu)]

I run it a 2nd time, it downloaded and installed r25 and all is fine now:

remains the question, what caused the update problem...

--
Sent from my Ubuntu phone
http://www.unixarea.de/

[Barry Warsaw (barry)]

I really don't know. Since channel.ini is created on the server when an image is published, it could have possibly been a server bug. I would expect other people to get hit by it too, and I think at least one other person was. That should be easy to verify by pulling down the data file from system-image.ubuntu.com for the device, channel, and version you got upgraded to, and unpacking it manually. The only other possibility is that something got corrupted in recovery, but that seems unlikely. I know of nothing else that touches channel.ini, and of course I'm assuming that you didn't edit it manually.

[Matthias Apitz (gubu)]

some remarks to Barry's last comment:

1) keep in mind that the device was nearly fully functional after the update, I only saw the problem because I went to the 'about' screen and because of the scopes; it could be the fact that more users are hit by the problem;

2) the channel.ini file is in a read-only file system; I have had to remount the root to correct it; i.e. it is nearly impossible that a normal user could damage that file;

3) I wanted today morning to check the image, but I only see small files in this place: http://system-image.ubuntu.com/ubuntu-touch/stable/bq-aquaris.en/krillin/ where is the full image?

thanks for all the support to get my wife's BQ working again!

[Matthias Apitz (gubu)]

please ignore the question 3; I learned how to use the ubuntu-device-flash tool and instelled it in my chrooted system in the BQ;

[Matthias Apitz (gubu)]

I have pulled the tar files for the device and channel and the file channel.ini is fine in the archive version-25.tar.xz;

more:
the file has in all three locations:
- my BQ r25
- the version-25.tar.xz
- the damaged file in my wife's BQ
the same last modification time 2 sep 11:33

it's absolutely unclear while it was of zero bytes in my wife's BQ; how the file is moved from the tar file in place during installation? it can not be a simple un-tar because the file in the tar archive is 'system/etc/system-image/channel.ini'

[Barry Warsaw (barry)]

On Sep 16, 2015, at 05:37 AM, Matthias Apitz wrote:

>1) keep in mind that the device was nearly fully functional after the
>update, I only saw the problem because I went to the 'about' screen and
>because of the scopes; it could be the fact that more users are hit by
>the problem;

If more people have the problem but don't know it yet, I'd expect a slew of
bug reports as people try to upgrade their phones. Have we seen any
indications of that yet?

>2) the channel.ini file is in a read-only file system; I have had to
>remount the root to correct it; i.e. it is nearly impossible that a
>normal user could damage that file;

Agreed! I knew it was a very remote possibility, but I wanted to make sure.

>3) I wanted today morning to check the image, but I only see small files
>in this place: http://system-image.ubuntu.com/ubuntu-touch/stable/bq-
>aquaris.en/krillin/ where is the full image?

Generally the data files in the cache directory are removed by recovery after
being applied. You'd have to reconstruct the urls and download them manually
from system-image.u.c to see what's in them.

[Simon Fels (morphis)]

On 16.09.2015 10:32, Matthias Apitz wrote:
> I have pulled the tar files for the device and channel and the file
> channel.ini is fine in the archive version-25.tar.xz;
>
> more:
> the file has in all three locations:
> - my BQ r25
> - the version-25.tar.xz
> - the damaged file in my wife's BQ
> the same last modification time 2 sep 11:33
>
> it's absolutely unclear while it was of zero bytes in my wife's BQ; how
> the file is moved from the tar file in place during installation? it can
> not be a simple un-tar because the file in the tar archive is
> 'system/etc/system-image/channel.ini'
>

It is a simple un-tar. Have a look at [1]. The system partition/image is
mounted at /recovery/system and the un-tar is run with the current
directory set /recovery which then completely matches the directory
layout in the update tarball. However it's a two-step process where
first all removed files are removed and then the new/changed files are
added/updated.

regards,
Simon

[1]:
https://code-review.phablet.ubuntu.com/gitweb?p=CyanogenMod/android_bootable_recovery.git;a=blob;f=system-image-upgrader;h=c645b0abebb2a2d1295fd20071a4f10b9ce78cf0;hb=refs/heads/phablet-4.4.2_r1#l515

[Matthias Apitz (gubu)]

I have checked the delta-files as well and I see no reason how the target file /etc/system-image/channel.ini could end up with length zero in my wife's BQ E4.5 after the update from r24 to r25 on September 13.

[Matthias Apitz (gubu)]

at the moment there is only one single case known of the issue; the other mentioned one was something else;

[Barry Warsaw (barry)]

I suspect that we're now seeing some of these in the error tracker:

https://errors.ubuntu.com/problem/eb68c18374c88008b38cc206f9e0b6fbb2ab0fe1

What's very odd is that neither the server or client has changed in months. What I don't know is whether some configuration on the server could be affecting this. Why si client should be more robust in the face of dangling symlinks, the server should very definitely not be generating tarballs with such dangling symlinks.

[Matthias Apitz (gubu)]

The above URL says: Sorry, you are not a member of a group that is allowed to see the data from error reports. Please fill out this form to request access.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Barry Warsaw (barry)]

I'm not really sure what the visibility is for the error tracker outside of Canonical, sorry. It's basically a page showing similar errors occurring out in the wild. All are for 15.10 systems and we've seen a smattering of reports dating back to May 2015, which corresponds roughly to the release of si 3.0. The fact that it's only showing up in Wily does explain why we're only seeing a few crashes here and there; there aren't any phones based on Wily by default yet.

What's odd is that in your original bug report, you identify the system as 15.04, and that should only have si 2.5, which cannot be vulnerable to this bug. So two mysteries remain:

How did you get si 3.0 on your system?

How did the server generate a dangling symlink?

[Matthias Apitz (gubu)]

to Barry:

I do not know what "si 2.5" or "si 3.0" is; I only can see what I have posted in the original report: version: Ubuntu 15.04-armhf (20150825-170536), can you please explain here (or off-list) what those "si values" are?

as I said, on the update to r25 on September 13, mine was working fine and my wife's failed; in my BQ a lot of things are installed below ~phablet, even a chroot complete system to compile things; the one of my wife is nearly unchanged (if I do not count contacts etc.); the only "unusual" with my wife's phone is that it was for "repair" in June here in Germany in the company Mundo Reader GmbH and I don't know what they did exactly as "repair"; it returned with a new MAC addr, that's why I think, they just returned another device which I then updated to r24; do you want me to contact Mundo Reader?

What do you mean with "dangling symlink"? I could not see any error in the fetched delta files nor in https://system-image.ubuntu.com/ubuntu-touch/stable/bq-aquaris.en/krillin/version-25.tar.xz ....

[Matthias Apitz (gubu)]

ah, si 3.0 stands for "system-image 3.0"... so, I do not know how this went to my wife's phone;

[Barry Warsaw (barry)]

On Sep 22, 2015, at 05:38 AM, Matthias Apitz wrote:

>I do not know what "si 2.5" or "si 3.0" is; I only can see what I have
>posted in the original report: version: Ubuntu 15.04-armhf
>(20150825-170536), can you please explain here (or off-list) what those
>"si values" are?

Yep, sorry, but glad you figured it out. :)

You can always find the version of the system-image client with:

$ system-image-cli --version

>as I said, on the update to r25 on September 13, mine was working fine
>and my wife's failed; in my BQ a lot of things are installed below
>~phablet, even a chroot complete system to compile things; the one of my
>wife is nearly unchanged (if I do not count contacts etc.); the only
>"unusual" with my wife's phone is that it was for "repair" in June here
>in Germany in the company Mundo Reader GmbH and I don't know what they
>did exactly as "repair"; it returned with a new MAC addr, that's why I
>think, they just returned another device which I then updated to r24; do
>you want me to contact Mundo Reader?

I'd find it *very* surprising if they reflashed the device with a wily-based
image. I wonder, is the chroot you mention wily or vivid-overlay? Is it at
all possible that when you set up the chroot, an update to system-image 3.0
leaked into the root partition? Again, that would be very surprising,
especially if your wife's phone was still read-only root.

>What do you mean with "dangling symlink"? I could not see any error in
>the fetched delta files nor in https://system-image.ubuntu.com/ubuntu-
>touch/stable/bq-aquaris.en/krillin/version-25.tar.xz ....

Dangling symlink means that there's a symbolic link that points to a file that
doesn't exist. Specifically, you *should* see something like this:

root@ubuntu-phablet:~# ls -l /etc/system-image/config.d/
total 0
lrwxrwxrwx 1 root root 13 Sep 21 18:59 00_default.ini -> ../client.ini
lrwxrwxrwx 1 root root 14 Sep 21 18:59 01_channel.ini -> ../channel.ini
root@ubuntu-phablet:~# ls -l /etc/system-image/
total 28
-rw-r--r-- 1 root root 1468 Oct 29 2014 archive-master.tar.xz
-rw-r--r-- 1 root root 836 Oct 29 2014 archive-master.tar.xz.asc
-rw-r--r-- 1 root root 252 Sep 20 23:25 channel.ini
-rw-r--r-- 1 root root 877 Oct 27 2014 client.ini
drwxrwxr-x 2 root root 4096 Sep 21 18:59 config.d
-rw-r--r-- 1 root root 4755 Sep 10 07:19 writable-paths

That's taken from my currently updated BQ phone:

root@ubuntu-phablet:~# system-image-cli -i
current build number: 129
device name: krillin
channel: ubuntu-touch/rc-proposed/bq-aquaris.en
last update: 2015-09-21 19:00:00
version version: 129
version ubuntu: 20150921
version device: 20150821-736d127
version custom: 20150821-887-33-32-vivid

root@ubuntu-phablet:~# system-image-cli --version
system-image-cli 2.5.1

All channels should have an /etc/system-image/config.d directory containing
two symlinks, with those symlinks pointing back into existing, non-zero files
in /etc/system-image. However, system-image 2.5.1 will ignore
/etc/system-image/config.d -- that directory is *only* relevant for
system-image >= 3.0, so even if those symlinks were dangling, si 2.5.1 would
never even notice.

I'm still not positive this is exactly the problem...

[Matthias Apitz (gubu)]

some small remarks:

1) the chroot in *my* phone is based on http://cdimage.ubuntu.com/ubuntu-touch/vivid/daily-preinstalled/current/vivid-preinstalled-touch-armhf.tar.gz and my wife's phone was never polluted with thing like this; she does not do Linux/UNIX;

2) we can just calm and wait for the next such incident and nail it down then;

thanks