New Unity Dash

[Michael Hall]

A new version of Unity's Dash, libunity APIs, the default and new "100 scopes", with a target date of April 1st for landing in Raring. The rest of Unity (Launcher, Indicators, HUD, etc) will not be affected.

For reference, the new "smart scope service" and corresponding Unity changes was discussed here: http://fridge.ubuntu.com/2013/01/30/smart-scopes/

The spec for the "smart scope service" is here: https://wiki.ubuntu.com/SmartScopes1304Spec

The goal is to have everything ready by April 1st. Until then, a ppa will be updated everyday for testing the feature by a group of 5 people (the ppa will be: https://launchpad.net/~ubuntu-unity/+archive/experimental-certified). Existing tests will be run as per of the daily release process. That + the manual testing, we'll have a good pick on the quality status before flipping the switch.

All packages will be prereviewed for NEWing and MIRing.

Code:

* Existing packages:
Unity:
lp:~unity-team/unity/trunk (will be merged in lp:unity)

Libunity:
lp:~unity-team/libunity/libunity-7.0 (will be merged in lp:libunity)

Apps scope:
lp:~unity-team/unity-lens-applications/libunity7-compatible

Files scope:
lp:~unity-team/unity-lens-files/libunity7-compatible

Gdrive scope:
lp:~submarine/unity-scope-gdrive/libunity-7-compatible

Gwibber scope:
pending Ken

Music scope:
lp:~unity-team/unity-lens-music/libunity7-compatible

Video scope:
lp:~unity-team/unity-lens-video/libunity7-compatible

China music scope:
lp:~ubuntukylin-members/unity-china-music-scope/unity-china-music-scope

* New packages:
Home scope:
lp:~unity-team/unity-scope-home/trunk (new package, will need a MIR as well as installed by default)

"100 Scopes":
New packages, will need MIR as well as installed by default:
lp:~submarine/ubuntu-scopes/tomboy
lp:~submarine/ubuntu-scopes/yahoostock
lp:~submarine/ubuntu-scopes/colourlovers
lp:~submarine/ubuntu-scopes/googlenews
lp:~submarine/ubuntu-scopes/calculator
lp:~submarine/ubuntu-scopes/yelp
lp:~submarine/ubuntu-scopes/imdb
lp:~submarine/ubuntu-scopes/deviantart
lp:~submarine/ubuntu-scopes/github
lp:~submarine/ubuntu-scopes/virtualbox
lp:~submarine/ubuntu-scopes/launchpad
lp:~submarine/ubuntu-scopes/soundcloud
lp:~submarine/ubuntu-scopes/manpages
lp:~submarine/ubuntu-scopes/gallica
lp:~submarine/ubuntu-scopes/devhelp
lp:~submarine/ubuntu-scopes/audacious
lp:~submarine/ubuntu-scopes/chromiumbookmarks
lp:~submarine/ubuntu-scopes/clementine
lp:~submarine/ubuntu-scopes/firefoxbookmarks
lp:~submarine/ubuntu-scopes/gmusicbrowser
lp:~submarine/ubuntu-scopes/guayadeque
lp:~submarine/ubuntu-scopes/musique
lp:~submarine/ubuntu-scopes/texdoc
lp:~submarine/ubuntu-scopes/openweathermap
lp:~submarine/ubuntu-scopes/gourmet
lp:~submarine/ubuntu-scopes/openclipart
lp:~submarine/ubuntu-scopes/sshsearch
lp:~submarine/ubuntu-scopes/themoviedb
lp:~submarine/ubuntu-scopes/zotero
lp:~submarine/ubuntu-scopes/evolution
lp:~submarine/ubuntu-scopes/sshsearch

This is the list I've been communicated as of 13.03.15

* Packages that we have to remove for the transition:
- unity-lens-shopping -> in the home scope service
- unity-lens-help -> some parts in the unity-scope-yelp and the onlin in the home scope service
- unity-lens-radios -> if not ported, nuke it
- unity-lens-vm -> in unity-scope-virtualbox
- unity-lens-sshsearch - in unity-scope-sshsearch
- unity-lens-pypi - has some performance issue, will probably just have to drop it

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1154229/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Scott Kitterman (kitterman)]

Why didn't this land before feature freeze?

[Benjamin Kerensa (bkerensa)]

I have the same question as ScottK.... We ran into this same issue when we approved a Feature Free Exception last release which landed the Amazon feature. Are we going to make exceptions to our development process every cycle?

[Jeremy Bícha (jbicha)]

Today is March 13th. Feature Freeze was March 7th. User Interface Freeze is March 21st. A "target date" of March 25th is unacceptable. Wasn't the daily build capability introduced this cycle for all raring components supposed to help with this?

A major problem with landing dramatic UI changes at or after UI Freeze is that there is no way anyone can fix the inevitable UI issues and inconsistencies without breaking UI Freeze even more. If an FFe is approved, this should last THIS WEEK so that the more obvious issues can be fixed by next week.

[Olli Ries (ories)]

re #2 & #3:

 it's a sabdfl request and we are working on the server/client integration atm. There was some ambiguity around the rolling release cadence and we were mislead to think that we would deliver into 13.04+n (n<6;). The plan state in the initial comment takes addresses quality concerns, landing by 3/25 would not be in the expected quality.

[Olli Ries (ories)]

re #4: is there a way to provide you with screenshots upfront or help the situation in another way

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Stéphane Graber (stgraber)]

If this is to be sabdfl'ed, the process is to have Mark explicitly say so as a comment here. Until then, don't consider this as approved in any way.

As I said on IRC, I really don't like the idea of this landing so late, especially as we've clearly said at UDS and in past FFe for Unity that we would be much stricter this time around. We even moved the FeatureFreeze a couple of weeks forward to try and reduce the likeliness of this happening. So I'm really surprised and disappointed that this new release couldn't be landed on time.

I can understand some of the confusion and delay caused by vUDS and some of the release cadence discussions and I may have been fine granting an exception if the feature was ready to land NOW, not in two weeks (which as Jeremy said, makes it past UI freeze).

[Scott Kitterman (kitterman)]

I think this is exactly the kind of update we agreed we would not approve when we moved feature freeze later in the cycle.

[Stefano Rivera (stefanor)]

Agreed. I thought that was well understood. FF was moved back to allow everything big to land before it.

[Steve Langasek (vorlon)]

Jeremy, the justification for UIFreeze has always been providing a stable target for documentation authors and translators, it is NOT because we believe there will be "inevitable UI issues" that require further freeze-breaking fixes. If the change is justifiable in terms of Feature Freeze, the UIFreeze should not be a barrier here - the change is being proposed in plenty of time for the documentation and translation teams to schedule their work accordingly.

That's not to say this should get a FFe - this is obviously a big change that needs to be carefully considered. But let's please not throw up additional roadblocks that don't follow from our freeze guidelines.

[Scott Kitterman (kitterman)]

Steve,

Although you are correct, I think it's reasonable to extrapolate from previous
experience and expect that the U/I in this next stack won't be final and so
screen shots would have to be redone at an indeterminate later point. To the
best of my recollection that's been pretty consistently the case when major
Unity updates were landed late in the release cycle.

[John Lea (johnlea)]

One thing to note is that if a FFE is granted for this change, the FFE will need to include covering the inevitable bugs that will be spotted and need to be fixed after this feature lands. Design has not yet been able to do any end to end testing of this feature, and after we start testing it is inevitable that UX bugs will be found that will need fixing.

The reason I am clarifying this is that I would like to avoid the problems we had last cycle, and make sure we are clear that landing this feature with a FFE also includes fixing the final UX bugs that will be spotted as soon as we start testing.

[Scott Kitterman (kitterman)]

The release team has already said no. It only gets in now on a sabdfl exception if one is provided.

[Didier Roche-Tolomelli (didrocks)]

@ScottK: I'm just updating the list with impacted components to have all infos on it. No need to be rude.

[Scott Kitterman (kitterman)]

Since when is setting correct bug status rude? I thought I was helping?

[Didier Roche-Tolomelli (didrocks)]

My remark was more on your comment than the bug status change "The release team has already said no.". Like if adding components was not needed.

But anyway, no need to bikeshed on this I guess :)

[Scott Kitterman (kitterman)]

Ah. That was done first as a reply to the previous comment and LP lumped it in with the later status changes.

[Olli Ries (ories)]

Scott, I have pinged Mark to weigh in. He is however travelling and it might be a while (Mo?) for his +1, jfyi.

[Mark Shuttleworth (sabdfl)]

Here's a +1 minus the 'explicit' language some have requested ;)

[Iain Lane (laney)]

So there we are.

Can you outline how this is planned to land please? Which packages will have to be uploaded, NEWed, MIRed and seeded?

Can I help this go more smoothly?

[Didier Roche-Tolomelli (didrocks)]

@Iain: see the description, I took great care in the previous days to try to answer that in advance, do you see information missing?

I've updated with today's list and progresses.

[Olli Ries (ories)]

Release team, we appreciate your support and would like to work with you as closely as possible to get the change in and enhance Ubuntu all together. I can understand the concerns about missed deadlines and am apologizing on behalf of the team for the troubled caused.

Here is what we think is happening and I would like to get your feedback to adjust things as quickly as possible:
- the team will take the ambiguity of the title "New Unity Stack" out by providing a better summary of affected components in order to help your assessment

- didrocks has set up PPAs to get test coverage in, first feedback is coming back already (based on the -prevalidation PPA). The PPAs are equivalent to the ones used for daily landing, except for pulling from a feature branch

- we will engage with Nicholas Skaggs to get additional test coverage

- while parts are ready to land now, the API changes render all dependencies unusable and we have been advised that it would be preferable to land everything in one go, rather in smaller iterations. Let us know your opinion

- the targeted landing on our end into the PPAs is 3/22

- I had offered help earlier to your team, if there is anything we can possibly do (screenshots?), let me know

Please note that the privacy settings as outlined in https://wiki.ubuntu.com/SmartScopes1304Spec will be landing in the PPA next week, the early version you can access today does not have all required pieces yet.

[Scott Kitterman (kitterman)]

According to the schedule, the archive will freeze for Beta 2 on March 28th:

https://wiki.ubuntu.com/RaringRingtail/ReleaseSchedule

I think it would be prudent to plan on having it all landed in the archive
prior to that time. That includes New reviews for new packages and rebuilds/
builds of reverse dependencies.

As there appear to be a non-trivial number of new packages involved, finding
archive administrators with the time to get proper reviews done can be a
challenge. I recommend you start work now to get people to block out time:

https://launchpad.net/~ubuntu-archive/+members#active

You'll need time for the packages to be uploaded, reviewed, any reject worthy
issue corrected, re-uploaded, re-reviewed, accepted, built, MIR'ed, and added
to the appropriate install media.

I think your offer on screen shots is most relevant to the docs team. I'd
recommend you contact them on their mailing list to discuss how you can
support their efforts.

[Didier Roche-Tolomelli (didrocks)]

@Scott: as written in the bug description, seb128 and I reshaped and fixed the packaging to make the "NEW" requirement. We did as well peer review on our changes to make that acceptable in ubuntu. So we do consider the NEWing part done and will do a finale NEWing check before landing that to the archive.

Does this address your concern?

We'll do the same with pre-MIR review. As usual when I'm doing the NEWing/involved in it, I won't be acking the MIR for it as it would be a conflict of interests.

My plan is when we get things in shape, to preping my MIR-team members fellow (mterry and doko) to pre-review the MIR.

[Didier Roche-Tolomelli (didrocks)]

Just a note: removing the shopping lens as the feature is baked in the home smart scope now. We'll just need to remove the shopping lens from the archive (the package transition is handled as well)

[Scott Kitterman (kitterman)]

As long as you've got it worked out, I'm happy. I mainly want to see this
landed before Beta 2 so we don't get forced into a lot of post-beta churn
again. It sounds like you've got most things covered.

[Olli Ries (ories)]

thanks Scott for your advice

[Nicholas Skaggs (nskaggs)]

We're lining up a call for testing to kick off first thing next week in anticipation of testing this. I'll report back with the milestone link so others can track the bugs found during this process.

[Thomas Strehl (strehl-t)]

Quick status update where we are. As of today we have
all default scopes ported (except gwibber/friends) and from a user point
of view we're almost back at 12.10 functionality. Biggest regression
against 12.10 is the filtering which we're currently fixing. In more detail:

- Scopes still to be ported for client side: pypi and evolution, estimated to
be done today

- Currently 9 scopes are run on server side: amazon, weather, synonym,
antonym, dictionary, duckduckgo_related, skimlinks, wikipedia and u1ms.
More to follow by 21.3.

- Doing a scope search in dash, server scopes are queried and results
are sent back to client. There results are properly inserted into categories
and can be activated. Smaller bugs here and there like missing icons etc.

- Server preview work is in progress, currently only the general
preview, but not music or any other specific one is working. Estimated
time of landing within couple of hours after getting proper support from
server. Petronis from server side already contacted but no estimate yet.

- Master scope filtering is incomplete as we're still missing some
filter attributes (definitions). Expected to be done by 21.3.

- Enabling/disabling of individual scopes via dash is biggest remaining
todo item. Estimated to land on 21. or 22.3.

[Nicholas Skaggs (nskaggs)]

Wanted to add the testing link for those who may not have seen it: http://packages.qa.ubuntu.com/qatracker/milestones/262/builds

[Michael Terry (mterry)]

I've been doing a pre-MIR review. Bug 1158782 (privacy support) should be fixed, but besides that things are mostly fine. Just nits, not blockers.

[Olli Ries (ories)]

After a recent review we have come to the conclusion that this feature is not at a stage that we'd like to see in Ubuntu and thus we are not pursuing to push this feature into Ubuntu 13.04.

Instead, we will be landing it in 13.10 once the current issues are resolved.
Ubuntu Users will be able to still use the feature by installing it from a Ubuntu R based PPA which will be rebased with any relevant R update. We will update this bug once the relevant details are documented.

In the course of the next couple of days we will also clean up this bug report with the necessary actions for R &S.

We appreciate your support and patience

[Kate Stewart (kate.stewart)]

Based on Ollie's comments above, marked this as nominated for S-Series.

[Steve Langasek (vorlon)]

If it's going to be landing in S, presumably it *won't* be landing after S feature freeze, so no FFe will be needed. So closing this bug report out instead.

[weihua zhang (whzhang-kylin)]

@Didier: Here is the source package of unity-china-music-scope with new libunity API, would like to sponsor us to upload it? Thanks.