ofono hitting an assert during settings test

[Ken VanDine]

We've been seeing some test failures in the cellular panel of system-settings, not always the same tests but similar symptoms. Looks like ofono hits and assert because the modem gets powered off. The common symptom in our tests is the various ofono related objects we get from libqofono become null and the syslog indicators that ofono exited. See attached syslog

https://jenkins.qa.ubuntu.com/job/generic-deb-autopilot-runner-vivid-mako/3204/testReport/junit/ubuntu_system_settings.tests.test_cellular/DualSimCellularTestCase/test_roaming_switch/

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

The reason for the crash seems to be an event that switches the radio state to UNAVAILABLE. This probably happens when rebooting the phone only, so it does not seem dangerous.

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

syslog from mako with the same syntomp.

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

https://github.com/rilmodem/ofono/pull/199 is a possible solution.

[Tony Espy (awe)]

The relevant messages from the syslog are:

Sep 1 07:55:45 ubuntu-phablet ofonod[1428]: ril_radio_state_changed: state: UNAVAILABLE rd->ofono_online: 1
Sep 1 07:55:45 ubuntu-phablet ofonod[1428]: ril_radio_state_changed: radio self-powered off!
Sep 1 07:55:45 ubuntu-phablet NetworkManager[1541]: <info> ofono disappeared

This is due to the fact that assert happens when the radio state is reported as UNAVAILABLE yet ofono online flag is TRUE. This indicates that radio has asynchronously powered off.

Bug #1479552 is an automatic bug created by crash reports seen at errors.ubuntu.com:

 /usr/sbin/ofonod:ERROR:plugins/ril.c:148:ril_radio_state_changed: assertion failed: (FALSE)

As such, I'll mark the other bug a duplicate of this one as we have at least two ways to duplicate this crash.

[Tony Espy (awe)]

@Ken

So although we have a pending fix that's been merged in our ofono trunk, I still would like to better understand what caused the crash during autopilot testing.

The fix removes an assert in the ril device plugin when the radio state becomes UNAVAILABLE and the plugin's online flag is TRUE. Essentially this assert is saying, if the radio ever goes OFF/UNAVAILABLE by itself without instruction from ofono, this is something that can't be handled... so assert, upstart restarts ofono, and in theory the system recovers.

That said, we never really thought thru what happens during system shutdown. As ofono is never told the system is shutting down, then whenever the container shuts down rild, it triggers the radio unavailable scenario and the assert happens, which could potentially lead to more than one crash/restart.

NOTE - with the proposed fix, UNAVAILABLE state transitions are *ignored*, however if OFF occurs instead, the same assert will still fire.

Can you or someone from your team isolate the AP test case that causes the failure? FlightMode shouldn't trigger this error as urfkill tells ofono to go offline, which triggers a RADIO POWER OFF message to rild. This sequence results in the plugin's online flag being set to FALSE *before* the radio power message is sent, so in theory, the assert should never happen.

If we have other non-shutdown scenarios that trigger the radio UNAVAILABLE state, ignoring the state may not be the wisest course of action.

Perhaps it's worth changing the assert to a graceful exit, returing a non-success error code, which would still allow upstart to restart ofono, but would no longer leave .crash files laying around.

Finally, we might also want to consider whether additional shutdown logic for ofono, urfkill, and other components in the networking/telephony stack is required. Note, there may be hooks and/or logic for shutdown in some of these components, however the timing might not be properly synced with the container shutdown.

[Ken VanDine (ken-vandine)]

@Tony

I think it's been found during a variety of tests, but I'll dig back through failures to look for common tests and get back to you.

[Łukasz Zemczak (sil2100)]

This bug was fixed in the package ofono 1.17.bzr6904+15.04.20150928.1-0ubuntu1 in https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/stable-phone-overlay

---------------

ofono (1.17.bzr6904+15.04.20150928.1-0ubuntu1) vivid; urgency=medium

  [ Alfonso Sanchez-Beato (email Canonical) ]
  * Update to upstream release 1.17
  * Do not assert when the radio is unavailable (LP: #1490991)
  * Fix crash when importing phonebook (LP: #1486004)
  * Check only destination port when receiving push (LP: #1490673)
  * Fix crash when retrying to close context (LP: #1492483)

  [ CI Train Bot ]
  * No-change rebuild.

ofono (1.16.bzr6902+15.10.20150911-0ubuntu1) wily; urgency=medium

  [ Alfonso Sanchez-Beato ]
  * New upstream release 1.16; Bluetooth disabled in prep for Bluez5.

  [ CI Train Bot ]
  * New rebuild forced.

ofono (1.12.bzr6900+15.10.20150724.1-0ubuntu1) wily; urgency=medium

  [ Alfonso Sanchez-Beato ]
  * Fix writing of EF_MWIS SIM file (LP: #1469266)
  * Improvement in DNS lookups to avoid the need to set routes for the
    DNS servers, by disabling the rp filter.

ofono (1.12.bzr6900+15.10.20150702.3-0ubuntu1) wily; urgency=medium

  [Tony Espy]
  * test/rilmodem/sim: fix testing scripts for arale (LP: #1457775)
  * test: reverse meaning of list-modems -p (LP: #1438715)

  [ Alfonso Sanchez-Beato ]
  * gril: cleanup traces
  * src/gprs.c: set preferred for the used IA APN (LP: #1361864)
  * rilmodem, mtkmodem, plugins/ril.c, plugins/mtk.c: retry when a
    context deactivation request has finished with an error
  * build, include, plugins/c-ares-dns-client.c, src/dns-client.c,
    src/gprs.c: resolve MMS proxy/MMSC host name (LP: #1417976)

  [ Ratchanan Srirattanamet ]
  * build, rilmodem, qcommsimmodem, plugins/ril.c,
    src/radio-settings.c: add multi-sim support for qcommsimmodem

ofono (1.12.bzr6896+15.04.20150521-0ubuntu1) vivid; urgency=medium

  [ Ratchanan Srirattanamet ]
  * build, gril, rilmodem, qcommsimmodem, plugins/qcom-msim.c: add
    support for LG L90 Dual which uses a Qualcomm dual-SIM modem (LP: #1427788)

  [ Alfonso Sanchez-Beato ]
  * src/gprs.c: support for 'ResetContexts' (LP: #1338758)
    Adds a new ConnectionManager 'ResetContexts' DBus method
    which deletes all existing gprs contexts and forces
    re-provisioning to occur.
  * plugins/ubuntu-apndb.c: ignore 'ipv6' protocol
    This change causes APNs with an 'ipv6' protocol value
    to instead be provisioned with 'ip', forcing IPv4 to
    be used.

  [ Jonas Drange ]
  * doc/connman-api.txt: add metion of 'ia' gprs context type

  [ Tony Espy ]
  * test/set-context-property: add 'Preferred' support (LP: #1454756, #1454751)
    This change adds support for the 'Preferred' ( and 'Active')
    properties which both need to be DBus variant wrapped booleans.
    Also added support for multi-SIM which was lacking from this script.
  * test/list-modems: add privacy support (LP: #1438715)
    Obfuscate any properties which can be tied to the end-user
    ( eg. SubscriberNumbers, SubscriberIdentity, ... ).

ofono (1.12.bzr6894+15.04.20150424.1-0ubuntu1) vivid; urgency=medium

  [ Tony Espy ]
  * rilmodem/gprs-context.c: notify if call-list is empty (LP...

[Launchpad Janitor (janitor)]

This bug was fixed in the package ofono - 1.17.bzr6904+15.10.20150928.1-0ubuntu1

---------------
ofono (1.17.bzr6904+15.10.20150928.1-0ubuntu1) wily; urgency=medium

  [ Alfonso Sanchez-Beato (email Canonical) ]
  * Update to upstream release 1.17
  * Do not assert when the radio is unavailable (LP: #1490991)
  * Fix crash when importing phonebook (LP: #1486004)
  * Check only destination port when receiving push (LP: #1490673)
  * Fix crash when retrying to close context (LP: #1492483)

  [ CI Train Bot ]
  * No-change rebuild.

 -- Alfonso Sanchez-Beato <email address hidden> Mon, 28 Sep 2015 09:27:18 +0000