Today scope refresh not always working on MX4 with OTA 13

[Dave H]

Hi I am using the Meizu MX4 with OTA-13 installed. Sometimes with weak or no wifi the today scope is messed up, in the same way as the below report I found for the BQE4.5. If you try to refresh it the problem is not resolved (such as only half the date shown, or no date or no pictures). Even after returning to a good wifi signal and performing the pulldown refresh the problem is not resolved. As I have seen this problem from the start on older OTA versions (I have reported before), my feeling is there is a fundamental problem with the way scopes are refreshed when you pull down the screen. I have also screen the "no pictures" on other scopes and the refresh not working, could this be a common scope problem and nothing to do with the today scope?.

Note, when the today scope is messed up if you un-tick the "day info" then refresh and then tick day info again, this sometimes helps. If no pictures displayed (today and other scopes) I normally have to reset the phone to correct the problem.

Your support us appreciated,

Regards

Dave H

Similar bug (BQE4.5 & OTA9)

* Upon updating today screen (swipe-down) get partially updated info, see attached step01.png
* Activating side bar/pressing on Ubuntu icon shows that the rest of scopes were updated, but they somehow ended up above today screen controls, see attached step02.png
* Activating side bar again makes it look like on attached step03.png
* Finally, pressing on Ubuntu icon once more results in empty today screen, step04.png

[Kyle Nitzsche (knitzsche)]

Hi Dave,

Sorry you are having an issue.

Can you please attach the screenshot that shows the issue, and provide the other bug number?

What version of the Today scope do you have.

And, right after the problem occurs, please get the following files and attach them:

/home/phablet/.cache/upstart/scope-registry.log
/home/phablet/.cache/upstart/unity8.log
/home/phablet/.cache/upstart/unity8-dash.log

Cheers

[Dave H (dhandhelen-s)]

Hi Kyle,
Thanks for getting back to me. The today scope version number is 4.7.0, according to the app history in ubuntu store (scope). It states this version is installed so I guess this must be correct. The OTA version I am using is OTA-13 (on both MX4 and BQ M10)

The problem is not as frequent as with earlier OTA versions but still happens when moving in and out of wifi areas or 3G only. What I have found is poor (slow) wifi seems to be the worst. The worrying part is once its messed up the pull down refresh wont clear it, you have to un-tick things in the today scope setting menu , refresh, then tick them again. A restart of the phone also clears the problem. The loss of the icon pictures also happens on other scopes on both my MX4 and BQ M10 tablet.
In this case only a restart clears this issue.

I will save a screen shot and the above directory content next time it occures (on both the MX4 and BQ M10 if possible). It may take a few day to capture on both.

Thanks for your support

Regards

Dave H

[Kyle Nitzsche (knitzsche)]

Hi Dave H,

I think you said there was another bug filed about this issue. (You seem to have copy/pasted from it in your bug description at the top).

It would be very helpful to know that bug number since it has attached images that you say show your problem.

Cheers

[Dave H (dhandhelen-s)]

Hi Kyle,

The older bug report was:

Bug #1546733 reported by Alexey Balmashnov on 2016-02-17

Looking at that I also reported still having a problem with OTA12 along with others, looks like you also got involved at the time.

The last entry from Anupam (anupam207) #29 , said "Oops, I spoke too soon! This started appearing again", so I guess he still has the problem.

Note, I have report problems with the Today scope before on older issues(including pictures). See below
Bug #1573996 reported by Dave H on 2016-04-23
Bug #1549983 reported by Dave H on 2016-02-25

Also I get the same as Thomas Feldmann (Bug #1546733), this is the missing or half cut dat icon, also still happening in OTA-13, sorry. Have a look at Thomas's picture attached in Bug #1546733
Thomas Feldmann (thomas-wiw) wrote on 2016-04-08: #21

    screenshot20160408_164551943.png Edit (116.6 KiB, image/png)

Thanks for checking this, once I have a good screen capture I will send it over.

Regards

Dave H

[Dave H (dhandhelen-s)]

Hi Kyle,
Have reproduced the problem again by walking away from my house. I have hand line wifi at 1.5M and wifi at 20M (via 3G wifi router) . With the phone connected to the 20M wifi router and increasing the distance until the signal is weak, things start to get mixed up.
First picture shows missing date and cut-off weather icons.

Will send the others one by one.

Regards

Dave

[Dave H (dhandhelen-s)]

Hi Kyle

Second shows just the date, even after refresh and moving to good wifi signal

[Dave H (dhandhelen-s)]

My today scope settings, weather, news and date should be displayed after refresh

[Dave H (dhandhelen-s)]

/home/phablet/.cache/upstart/scope-registry.log

[Dave H (dhandhelen-s)]

/home/phablet/.cache/upstart/unity8.log

[Dave H (dhandhelen-s)]

/home/phablet/.cache/upstart/unity8-dash.log

Note, one strange thing, after the above my BBC news scope is showing no icons or text even after refresh. Other scopes such as Euronews refresh OK. Not sure if this is related.

I checked my wifi connection and its 16~20M with strong signal (using FAST app and Speedofme lite). The internet also works fine, but the today scope and BBC news scope wont refresh correctly and stay broken. If I remember I have also seen this with other scopes.

After resetting the phone all is back to normal.

Hope all this helps, would be great to resolve this issue before OTA-14 is released.

Regards

Dave H

[Dave H (dhandhelen-s)]

Sorry forgot to say, when I was outside house at some distance from my two wifi routers, I may have switched between (short time on 4G) them both before reproducing the problem. I also turned the data and weather info on/off in the Today scope setting a few minutes before reproduced the problem. Once the problem was seen I only tried to refresh, first at long distance, then close to the 20M speed router, with no effect.

Regards

Dave H

[Kyle Nitzsche (knitzsche)]

Hi David H.

So, a few things which I will address in separate comments for clarity.

First, your scope-registry.log contains a unity::scopes::MiddlewareException

[2016-11-02 17:04:35.351] ERROR: com.canonical.scopes.dashboard_dashboard: ReplyImpl::error(): unity::scopes::MiddlewareException: Cannot invoke operations while middleware is stopped

Whenever I have seen this, it is a Game Over situation usually requiring a reboot.

I am not sure what causes this, but it is part of the unity-scopes-api. I suspect it is a bug there. so I will add unity-scopes-api package here.

[Kyle Nitzsche (knitzsche)]

David H.

In one of your other bugs about Today scope display issues [1], I think you mentioned that you made the rootfs "/" read-write (in order to change the default scope background as I recall).

Doing this is inherently dangerous and can cause all kinds of problems that are essentially impossible to track down and are not real. The problem is that over-the-air updates are applied as diffs to files. Any file that has been modified and that gets an update is thereafter broken with no recovery path short of reflashing the rootfs. I am not saying that this is the cause of this bug. I am saying that by modifying the rootfs you open the door to this possibility.

[1] https://bugs.launchpad.net/today-scope/+bug/1549983/comments/4

[Kyle Nitzsche (knitzsche)]

David H,

Assuming you have not modified any other parts of your rootfs, let's get you the latest Today scope and see if you can reproduce any of the issues, just to be sure.

Please install the attached and reboot (just to get the logs cleared) and comment back with further info.

[Kyle Nitzsche (knitzsche)]

And provide the scope-registry.log if the issue is reproduceable with that recent today scope. (I am interested to see if the MiddlewareExecption occurs again).

[Kyle Nitzsche (knitzsche)]

I was just able to produce a MiddelwareException twice by walking out of range of my wifi while a Today scope refresh was in progress.

In both case the resulting display ONLY the Day scope.

In one case there was also a twitter scope panic/crash.

Attached my scope-reg.log

[Kyle Nitzsche (knitzsche)]

After walking back into wifi range (on my second reproduction) and refreshing Today scope, only Day scope displays (although it does display normally).

The scope-reg.log contains this now reliably on refresh with only Day scope showing:
[2016-11-03 16:56:46.984] INFO: Registry: RegistryObject::ScopeProcess::exec(): Process for scope: "com.canonical.scopes.dashboard_dashboard" started
"com.canonical.scopes.dashboard_dashboard: ADDING DECLARED child scope: com.canonical.scopes.day_day, local_id: day_localId"
"com.canonical.scopes.dashboard_dashboard: ADDING DECLARED child scope: com.canonical.scopes.weatherchannel, local_id: weather_channel_localId"
"com.canonical.scopes.dashboard_dashboard: ADDING DECLARED child scope: com.canonical.scopes.holidays_holidays, local_id: holidays_localID"
"com.canonical.scopes.dashboard_dashboard: ADDING DECLARED child scope: com.canonical.scopes.events_events, local_id: events_localID"
"com.canonical.scopes.dashboard_dashboard: ADDING DECLARED child scope: com.canonical.scopes.tasks_sctasks, local_id: tasks_localID"
"com.canonical.scopes.dashboard_dashboard: ADDING DECLARED child scope: com.canonical.scopes.contacts_contacts, local_id: contacts"
"com.canonical.scopes.dashboard_dashboard: ADDING DECLARED child scope: com.canonical.scopes.fitbit_fitbit, local_id: fitbit_localId"
"com.canonical.scopes.dashboard_dashboard: ADDING CATEGORY KEYWORD child scope: com.canonical.scopes.calls_calls, by keyword recent, to department: "
"com.canonical.scopes.dashboard_dashboard: ADDING CATEGORY KEYWORD child scope: com.canonical.scopes.texts_texts, by keyword recent, to department: "
"com.canonical.scopes.dashboard_dashboard: ADDING CATEGORY KEYWORD child scope: com.ubuntu.telegram_sctelegram, by keyword recent, to department: "
"com.canonical.scopes.dashboard_dashboard: ADDING CATEGORY KEYWORD child scope: com.canonical.elpais_ELPAIS, by keyword news.headlines, to department: "
"com.canonical.scopes.dashboard_dashboard: ADDING CATEGORY KEYWORD child scope: com.canonical.scopes.bbc_bbc, by keyword news.headlines, to department: "
"com.canonical.scopes.dashboard_dashboard: ADDING CATEGORY KEYWORD child scope: com.canonical.scopes.eljueves_eljueves, by keyword news.headlines, to department: "
"com.canonical.scopes.dashboard_dashboard: ADDING CATEGORY KEYWORD child scope: com.canonical.scopes.euronews_euronews, by keyword news.headlines, to department: "
"com.canonical.scopes.dashboard_dashboard: ADDING CATEGORY KEYWORD child scope: com.canonical.scopes.testscope_testscope, by keyword news.headlines, to department: "
"com.canonical.scopes.dashboard_dashboard: ADDING CATEGORY KEYWORD child scope: knitzsche.testscope-art_testscope-art, by keyword news.headlines, to department: "
"com.canonical.scopes.dashboard_dashboard: ADDING CATEGORY KEYWORD child scope: knitzsche.testscope-emblem_testscope-emblem, by keyword news.headlines, to department: "
"com.canonical.scopes.dashboard_dashboard: ADDING CATEGORY KEYWORD child scope: knitzsche.testscope-mascot_testscope-mascot, by keyword news.headlines, to department: "
"com.canonical.scopes.dashboard_dashboard: ADDING CATEGORY KEYWORD child scope: twitter.canonicalpartners_twitter, by ke...

[Kyle Nitzsche (knitzsche)]

Noting the TimeoutException on holidays scope above, I've tried launching holidays from Manage page. No results display.

running holidays scope from command line with scope-data tool I get this with no displayed results:
phablet@ubuntu-phablet:~$ ./scope-data com.canonical.scopes.holidays_holidays --results
unity::scopes::TimeoutException: Request timed out after 500 milliseconds (endpoint = ipc:///run/user/32011/zmq/priv/com.canonical.scopes.holidays_holidays, op = debug_mode)

I will now try reproducing the Today scope bug with Holidays scope disabled.

[Kyle Nitzsche (knitzsche)]

by the way, issuing "restart scope-registry" brings holidays scope back to life and fixes the today scope on refresh.

[Kyle Nitzsche (knitzsche)]

ok, with holiday scope disabled, I walk out of wifi range while a today scope refresh is ongoing and I get similar results:
* only the Day scope displays
* but this time a different child scope displays the TimeoutExecption: elpais

[2016-11-03 17:12:03.323] ERROR: com.canonical.scopes.dashboard_dashboard: QueryBase::run(): unity::scopes::TimeoutException: Request timed out after 500 milliseconds (endpoint = ipc:///run/user/32011/zmq/priv/com.canonical.elpais_ELPAIS, op = debug_mode)

again, restarting scope-registry fixes today scope.

[Kyle Nitzsche (knitzsche)]

twice now with elpais and holidays disabled in Today during refresh I walked out of wifi range and *expected* resulted displayed [1]. Display was not limited to Day scope.

Possibility: these child scopes do not handle loss of network during refresh correctly. They should simply return from the run() method.

Does the unity::scopes::QueryMetadata::ConnectivityStatus [2] update dynamically during a scope run() exuction, or does it just have a value that was correct when first passed to the scope?

[1] there was also a twitter scope panic. I filed this bug for that: https://bugs.launchpad.net/twitter-scope/+bug/1639056

[2] https://developer.ubuntu.com/api/scopes/cpp/sdk-15.04.4/unity.scopes.QueryMetadata/#a20eb916661728a7d9c00485e28f88701

[Kyle Nitzsche (knitzsche)]

Given my above experiences, and given that you do have elpais installed and it may be enabled in your Today scope (can't see from your settings image in comment #7), can you please:

Ensure that El Pais and holidays scopes are DISABLED in today scope settings, and see if you can reproduce this? (if you have the twitter scope installed, disable that too).

Thanks for your patience!

[Michi Henning (michihenning)]

Thanks for all the detective work!

I suspect that the MiddlewareException in coincidental and caused by an earlier timeout. But we shouldn't get stuck like this and things should recover transparently if the network goes away. I'll have a look and see what's going on.

[Dave H (dhandhelen-s)]

Hi Kyle,

Thanks for all your efforts to try to resolve this issue. I have install the file you sent using the below method:

Logged onto my MX4 in developer mode using adb shell (in read only mode). Copied your file to the phone Download directory and:

phablet@ubuntu-phablet:~/Downloads$ pkcon --allow-untrusted install-local com.canonical.scopes.dashboard_4.11.1_armhf.click

The package installed OK! (I am no software engineer, but will have a go)

Have just tried the New Today scope version 4.11.1, please see below result.
Long distance weak wifi - on refresh on the data shown.
Moving into just 3G - after a few refreshes first the data then weather and BBC shown.
Moving back into weak wifi - on refresh just date icon shown (no weather or BBC)
Back to my house with strong wifi - still only date shown even after 5 pulldown refresh actions.
After more refreshes the screen went blank.
Checked my wifi speed with FAST app , OK about 24M
Checked internet - all OK

Looking at the Today scope again, refresh sometimes shows the date icon only or blank screen.

Entered Today scope settings and turned off weather and BBC (I dont have holidays selected and cant see any elpais option on my phone). My selected items in the Today scope settings are listed below.
Location data
Contacts
Day Info
Weather Channel
BBC
euronews

Now on refresh the euronews pictures are back, sometimes!
I then turned the BBC and weather back on (in Today scope settings)- strange, sometimes I get the weather other times just the date icon.

From this point, no number of refreshes could correct the today scope so I reset the phone.
All then back to normal.

Hope this helps

Regards

Dave H

[Kyle Nitzsche (knitzsche)]

Dave - thanks.

My guess is this has nothing to do with the Today scope.

I guess is that there is something in the plumbing that does not handle losing a network during a fresh - so I am glad that Michi is aware.

Regarding network, I wrote and attached here a test scope that displays the scopes current understanding of the network status twice, the second time is after waiting 10 seconds.

The code is here: lp:~knitzsche/+junk/test-network-dynamic-enum

I find that the unity::scopes::QueryMetadata::ConnectivityStatus is wrong much of the time. It is often Unknown for example, whereas I have been sitting here at my desk with connected wifi all along.

[Dave H (dhandhelen-s)]

Hi Kyle,

Thanks for the Test scope. I installed it and found a similar result to you. If you keep repeating the refresh every 30s (ten times in my case) you get a mixed result.

1. 1 unknown / 2 unknown

2. 1 unknown / 2 unknown

3. 1 connected / 2 unknown (after pull refresh in Today scope and recheck Test scope)

4. 1 connected / 2 unknown

5. 1 unknown / 2 unknown

6. 1 connected / 2 unknown

7. 1 unknown / 2 unknown

8. 1 unknown / 2 unknown

9. 1 connected / 2 unknown

10. 1 unknown / 2 unknown

See picture attached (for 1 connected / 2 unknown)

If you need me to try anything else let me know.

Regards

Dave H

[Michi Henning (michihenning)]

The connectivity status is something that is *not* set by scopes-api. Instead, the status is received by scopes-api from whoever makes the query (typically, the shell) and then passed through to the scope. In turn, the shell just passes on what it gets from QNetworkAccessManager, I believe.

We had bugs open against QNAM for this and there were some changes made, but it's still flaky.

At any rate, the connectivity status is unlikely to be useful in debugging this issue because scopes-api never pays attention to it.

[Dave H (dhandhelen-s)]

Hi Michi,

OK thanks for checking and I agree with your explanation as you can see similar problems with other scopes. I guess the Today scope looks worse (and more annoying) as its the main scope you see each time the phone is active, others are not always checked or used. Also the scopes with multiple feeds are more effected. This problem wouldn’t be so bad if they cleared them selves once a good wifi signal returns, but as it is now you sometimes have to restart the phone. If this could be resolved I think it would make Ubuntu Touch look more stable for normal every day use, so should be given a high priority for OTA-14 / 15.

Note, have attached this mornings Today scope crash. In this case the wifi has been active with good signal all night. Looking at the settings in my 4G EE home broadband router, the DHCP lease time is set to 86400s (1 day) so the IP number could have changed overnight, note sure if this could also be related to general wifi issues.

Thanks for your continued support and hard work,

Best regards

Dave H

[Dave H (dhandhelen-s)]

Hi Kyle and Michi,

Just to let you know, I still have the problem with the Today scope refresh / messed up icons, on my MX4 with OTA-14 installed. I was wondering if you had any time to look again at this problem.

Merry Christmas and happy new year to you all!

Best regards

Dave H