Urfkill saved wrong WWAN state after enabling/disabling flight mode

[Ricardo Salveti]

current build number: 247
device name: krillin
channel: ubuntu-touch/ubuntu-rtm/14.09-proposed
last update: 2015-02-27 04:59:10
version version: 247
version ubuntu: 20150227
version device: 20150225-b67e0b6
version custom: 20150216-561-29-186

After enabling/disabling flight mode, the urfkill state for WWAN got in a broken state making me unable to get ofono to work anymore (not even after trying to enable flight mode again, or even rebooting the phone).

Here is what I did when I got this bug:
1 - Carrier signal was really poor, making ofono to start searching for the carrier again (could see 'searching' in the indicator).
2 - Not connected to any WIFI AP (but wifi was enabled).
3 - After not being able to automatically connect to my carrier's network, I decided to enable/disable flight mode while it was still searching for the network.
4 - That made ofono to be in offline mode, and never really being able to enable it again (not even after reboots).

Looking at the urfkill state file, it seems WWAN was soft=true even when flight mode was not enabled anymore, which could explain the reason why I'm not able to register anymore not even between reboots.

[Ricardo Salveti (rsalveti)]

phablet@ubuntu-phablet:/usr/share/ofono/scripts$ ./list-modems
[ /ril_1 ]
    Manufacturer = Fake Manufacturer
    Online = 0
    Revision = MOLY.WR8.W1315.MD.WG.MP.V37.P5, 2014/05/15 11:49
    Lockdown = 0
    Features =
    Interfaces = org.ofono.CallVolume org.ofono.VoiceCallManager org.ofono.NetworkTime
    Type = hardware
    Model = Fake Modem Model
    Emergency = 0
    Serial = 354142060305523
    Powered = 1
    [ org.ofono.CallVolume ]
        Muted = 0
        MicrophoneVolume = 0
        SpeakerVolume = 0
    [ org.ofono.VoiceCallManager ]
        EmergencyNumbers = 08 000 999 110 112 911 118 119
    [ org.ofono.NetworkTime ]

[ /ril_0 ]
    Manufacturer = Fake Manufacturer
    Online = 0
    Revision = MOLY.WR8.W1315.MD.WG.MP.V37.P5, 2014/05/15 11:49
    Lockdown = 0
    Features =
    Interfaces = org.ofono.CallVolume org.ofono.VoiceCallManager org.ofono.NetworkTime
    Type = hardware
    Model = Fake Modem Model
    Emergency = 0
    Serial = 354142060025527
    Powered = 1
    [ org.ofono.CallVolume ]
        Muted = 0
        MicrophoneVolume = 0
        SpeakerVolume = 0
    [ org.ofono.VoiceCallManager ]
        EmergencyNumbers = 08 000 999 110 112 911 118 119
    [ org.ofono.NetworkTime ]

phablet@ubuntu-phablet:/usr/share/urfkill/scripts$ ./flight-mode
flight-mode is: 0

phablet@ubuntu-phablet:/usr/share/urfkill/scripts$ ./enumerate
/org/freedesktop/URfkill/devices/0
Printing props for unspecialized device: /org/freedesktop/URfkill/devices/0
Printing props for Kernel device: /org/freedesktop/URfkill/devices/0
index: 0
platform: 0
urftype: org.freedesktop.URfkill.Device.Kernel
type: 2
name: hci0
hard: 0
soft: 0

/org/freedesktop/URfkill/devices/200
Printing props for unspecialized device: /org/freedesktop/URfkill/devices/200
Printing props for Hybris device: /org/freedesktop/URfkill/devices/200
index: 200
platform: 0
urftype: org.freedesktop.URfkill.Device.Hybris
type: 1
name: hybris_wifi
soft: 0

/org/freedesktop/URfkill/devices/100
Printing props for unspecialized device: /org/freedesktop/URfkill/devices/100
Printing props for Ofono device: /org/freedesktop/URfkill/devices/100
index: 100
platform: 0
urftype: org.freedesktop.URfkill.Device.Ofono
type: 5
name: Fake Manufacturer Fake Modem Model
soft: 1

/org/freedesktop/URfkill/devices/101
Printing props for unspecialized device: /org/freedesktop/URfkill/devices/101
Printing props for Ofono device: /org/freedesktop/URfkill/devices/101
index: 101
platform: 0
urftype: org.freedesktop.URfkill.Device.Ofono
type: 5
name: Fake Manufacturer Fake Modem Model
soft: 1

[Ricardo Salveti (rsalveti)]

phablet@ubuntu-phablet:/var/lib$ cat urfkill/saved-states
[WLAN]
soft=false
prev-soft=false

[BLUETOOTH]
soft=false
prev-soft=false

[UWB]
prev-soft=false
soft=false

[WIMAX]
prev-soft=false
soft=false

[WWAN]
prev-soft=false
soft=true

[GPS]
prev-soft=false
soft=false

[FM]
prev-soft=false
soft=false

[NFC]
prev-soft=false
soft=false

[ALL]
soft=false

[Ricardo Salveti (rsalveti)]

This might be a similar to bug 1321627

[Ricardo Salveti (rsalveti)]

Stopping urfkill, removing state file and rebooting made it to work again

[Ricardo Salveti (rsalveti)]

Tried for more than 20 times and can't reproduce the problem. Seems to be a hard one to reproduce.

[Tony Espy (awe)]

I also assisted Julian Funk today with the same problem. This time on mako/RTM. Changed urfkill task to Confirmed accordingly.

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

https://github.com/alfonsosanchezbeato/urfkill/tree/strict-fm

is a solution to make sure that even when there are problems we can return to a valid state after a reboot.

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

It is difficult to reproduce this, I was able to arrive to a bad state following these steps:

1. Boot system with 2 SIMs, normal state
2. Offline both modems using ofono scripts
        $ cd /usr/share/ofono/scripts
        $ ./offline_modem /ril_0
        $ ./offline_modem /ril_1
3. Toggle on flight mode
4. Toggle off flight mode. The modems are not onlined, and soft=true for WWAN, but not for ALL in saved-states file.

urfkill is keeping the previous state of the modems (the state before FM was set). I wonder if for some reason/bug the modems were offline when the bug happened. The "searching" status could be for one modem, but the other could be offline, which is enough for triggering the bug. Or, the searching status was wrong and the modem was actually offline.

This PR makes WWAN state == flight mode (ALL) state, fixes the situation described in this comment and as a minimum would let us recover with a reboot in case we come across again with a "saved-states" with content as in comment #2:

https://github.com/cyphermox/urfkill/pull/19

[Leo Arias (elopio)]

Alfonso, the linked pull request has no regression test to confirm the fix for the problem you described. Can you write a test for it?

Also, from what you say about the searching status, it sounds like there should be better logging to know which modem is searching.

[Tony Espy (awe)]

@Leo

What this change does is ensure that when "strict flight-mode" is enabled ( which is the default ), that the state of the modem will always follow the global state of flight-mode itself.

The only way to test this is to stop urfkill, and modify its saved-state file so that it contains "[WWAN] soft=true", reboot the phone, and ensure that the modem is active.

I discussed with Alfonso, and he'll add this as a new case to the exiting urfkill test plan.

I'd also like to note that this is a safeguard that will allow the user to recover their device if the original problem is hit again. It's not a total solution, so we should make sure not include the bug in the debian/changelog, as there's more work to do on this one...

[Leo Arias (elopio)]

Tony, a manual test is good, so thanks.

An automated test is even better. We are now working on the suite to test upgrades, and we learned about ways to do a reboot of the test bed using adt-run. So what you describe is automatable, but it's up to you to decide if it's worth it or it's better to keep it manual. Let us know if you are interested in adt-run reboots and we'll give you a hand.

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

Test 9 added to urfkill test plan:

https://wiki.ubuntu.com/Process/Merges/TestPlans/urfkill

[Ricardo Salveti (rsalveti)]

We need to land this in both vivid and RTM.

[Julien Funk (jaboing)]

Until the fix is released...

After running into this problem the following steps got my phone back up and running (Nexus 4, r214):

1) Open the terminal app on the phone and validate using password if configured

2) $ sudo stop urfkill

3) $ sudo rm /var/lib/urfkill/saved-states -rf

4) $ sudo reboot

The phone should come back up with the carrier re-enabled.

[Launchpad Janitor (janitor)]

This bug was fixed in the package urfkill - 0.6.0~20150318.103828.5539c0d.1-0ubuntu1

---------------
urfkill (0.6.0~20150318.103828.5539c0d.1-0ubuntu1) vivid; urgency=medium

  * New upstream snapshot, bugfix only:
    - Guard against double call to set ofono's online property.
    - UrfConfig: make the strict flight mode a configuration parameter.
    - UrfConfig: Enforce strict flight mode when set. (LP: #1427439)
 -- Mathieu Trudel-Lapierre <email address hidden> Thu, 19 Mar 2015 20:50:13 -0400

[Pat McGowan (pat-mcgowan)]

tagging to land in the update

[Launchpad Janitor (janitor)]

This bug was fixed in the package urfkill - 0.6.0~20150318.103828.5539c0d.1~rtm-0ubuntu1

---------------
urfkill (0.6.0~20150318.103828.5539c0d.1~rtm-0ubuntu1) 14.09; urgency=medium

  * New upstream snapshot, bugfix only:
    - Guard against double call to set ofono's online property.
    - UrfConfig: make the strict flight mode a configuration parameter.
    - UrfConfig: Enforce strict flight mode when set. (LP: #1427439)
 -- Mathieu Trudel-Lapierre <email address hidden> Thu, 19 Mar 2015 20:50:13 -0400

[Tony Espy (awe)]

I've re-opened the urfkill (Ubuntu) task as the underlying bug can still occur, and the fix released is a workaround which will allow a user to recover their modem by rebooting the phone.

We still have not managed to reproduce the underlying problem. What's needed is a stress test script for urfkill which can be configured to run for hours. Something similar to the checkbox reboot, suspend, hibernate tests used by Canonical's mainstream HW enablement team.