[FFE] Update WebApps to support dpkg distribution mechanism

[Alexandre Abreu]

A new integration script mechanism has been put in place (both for "local" integration scripts installed and ones pulled from "outside" through a debian package), the associated WebApps extensions need to be updated to support this.

The extent of the work is strictly confined to the respective WebApps browser extensions that will pull information from libunity-webapps about the integration script(s) associated with a given visited url.

This change is necessary to properly support external integration script community support.

[Kate Stewart (kate.stewart)]

What testing has occured on this? When can it land?

Has the security team reviewed this approach?

[Marc Deslauriers (mdeslaur)]

Is there a diff we can review to see what the security implications of this is?

[Robert Carr (robertcarr)]

A fair amount of testing has occurred, the code actually landed prior to FFE but wasn't fully enabled, this is some bug fixing around it. Some of it is also covered by tests, or is a derivative of older tested code.

There is not an exact diff, but the security implementations as far as this goes are all contained in lp:libunity-webapps src/libunity-webapps-repository. The extensions interact with this library through the UnityWebappsApplicationRepository object contained in unity-webapps-application-repository.h.

[Ken VanDine (ken-vandine)]

The aptdaemon end of this is handled under bug 1035207

[Iain Lane (laney)]

Do I understand correctly that this is about the browser triggering the installation of Debian packages from the Ubuntu repository when visiting webapps-compatible websites?

It looks like the firefox support is not too close to landing (bug #1040313). Is there much point to landing this without that?

I'd like security to give their approval to this first.

Setting to Incomplete, please reset to New once you've answered these questions (if the answers to the last two are affirmative).

[Alexandre Abreu (abreu-alexandre)]

Yes for the two questions,

We are in discussion for the Firefox support issues, but landing this also matters in order to have chromium working,

[Alexandre Abreu (abreu-alexandre)]

A few comments to help clear things out:

- this is a key feature of the WebApps technology that would actually prevent it from properly working so its "Importance" should be set to 'high',

- the code is strictly confined to libunity-webapps and has been tested for some time now, as Robert Carr pointed out most of it is covered by tests,

- the feature is actually split into two different components: when visiting a website, libunity-webapps searches for matching "integration scripts" first locally, looking for "preinstalled" integration scripts (associated with specific web applications) in /usr/share/unity-webapps/userscripts that simply consists in plain javascript files along with a basic manifest, if nothing is found, it searches for available packages (pulled on a regular basis from the Ubuntu repo) and installs it. Those packages only consists of a set of icons and javascript files being after considered by the WebApps extension.

[Kate Stewart (kate.stewart)]

If this causes issues, what is the plan for reverting? what dependencies need to be taken into account?
Can you please add a code diff so security team has somthing to review?

[Alexandre Abreu (abreu-alexandre)]

- in case of issues, we have a few options:

    * remove the code that tries to install a package associated with an integration script that has been found to be "available but not installed" (i.e. a package exists for it) for a given URL, the code would be removed w/ a low & controlled risk from the chromium and firefox extensions,

    * in libunity-webapps-repository, when searching for an available integration script for a given URL, two "sources" are being considered: scripts that are found locally (already installed previously using the installation process described above, or installed through a package such as lp:webapps-applications with an --enable-applications=yes flag), and scripts that are "installable" e.g. found in apt-cache as WebApps packages and for which an "installation" process can be launched. The two sources can be considered in that order and can therefore be disabled individually there: http://bazaar.launchpad.net/~webapps/libunity-webapps/trunk/view/head:/src/libunity-webapps-repository/unity-webapps-application-repository.c in the function unity_webapps_application_repository_resolve_url,

- the code can be browsed in src/libunity-webapps-repository and src/index-updater http://bazaar.launchpad.net/~webapps/libunity-webapps/trunk/files/753/src/libunity-webapps-repository and http://bazaar.launchpad.net/~webapps/libunity-webapps/trunk/files/753/src/index-updater (see breakdown below),

- since there seem to be some (legitimate) confusion about how it works here is a quick walk-through (sorry for the length):

Part of the WebApps technology, integration scripts that expand the "scope" of a given website (URL) are being "run" along with a given webpage by dedicated extensions (for chromium and firefox). Those scripts (small snippets of code, a few 10s of line of code) basically create a bridge between selected elements of the desktop (currently messaging indicator, launcher, sound menu, hud, notifications) and available resources in the website itself.

We have a current set of default scripts (around 40), the idea is to allow the community to also participate to the effort and contribute to new integration scripts.

The integration scripts will be made available as individual packages (deb) each one containing a set of icons, some doc, and the core javascript file itself. They will be made available through our repos (after going through a very strict review phase).

Along with each integration script, a manifest file ("manifest.js") will also be found that declares a set of descriptive meta-information about the WebApp itself: a glob friendly URL for which it should be triggered as an integration point, a homepage for the application, names of icons associated with the application (installed), a description, various version numbers, etc.

Now, coming back to the core functionality itself. When a website is being visited (URL), the libunity-webapps-repository lib is being queried by the corresponding browser extensions to check whether an integration script is associated to it. The entry points for the query itself are located in the "unity-webapps-application-repository.c" file in the unity_webapps_application_rep...

[Marc Deslauriers (mdeslaur)]

Thanks very much for the explanation, that definitely clears up a lot of questions I had. I wish the README file could be filled out in later versions with some of the details on what libunity-webapps is, and how it works.

I see in unity-webapps-apt-package-mechanism.c that it uses PackageKit to do the actual package installation, which is good as I didn't want libunity-webapps to silently install packages.

I believe the changes are ok security-wise, and I'm confident that the team will quickly address any security issues that are subsequently discovered, if any.

ACK.

[Kate Stewart (kate.stewart)]

ACK from release team, please get this in before beta 2 freeze if possible.

Please make sure to update the README and provide content to the Release Notes for this.

[Jamie Strandboge (jdstrand)]

I realize that this bug is about libunity-webapps and not about the browser extensions, but this bit concerns me:
"Part of the WebApps technology, integration scripts that expand the "scope" of a given website (URL) are being "run" along with a given webpage by dedicated extensions (for chromium and firefox)."

What guarantees do we have that the extension won't break in the next browser security update? Are there tests (preferably automated) that can be run to verify that the new browser works with the old extension? Who will be doing this testing? Who is maintaining the browser extension code in stable releases? I have quite a few concerns here because these updates come every 3-6 weeks and we don't want to have to decide between providing a timely, important security update to our users and breaking the browser extension (and therefore webapps).

[Alexandre Abreu (abreu-alexandre)]

- "What guarantees do we have that the extension won't break in the next browser security update?"

The extensions all use each browser's built-in extension mechanism, we don't rely on specific mechanisms besides that only the "public" browser extension api. We have (will) to keep track of the changes in this area to make sure that nothing breaks, (and update what's needed if it does),

- "Are there tests (preferably automated) that can be run to verify that the new browser works with the old extension?"

We have automated integration tests that are being run daily to make sure that basic integration happens. These are based on autopilot's desktop introspection caps,

- "Who will be doing this testing?"

See above, our QA team does take care of maintaining (and expanding) them,

- "Who is maintaining the browser extension code in stable releases?"

The WebApps team does,

- "I have quite a few concerns here because these updates come every 3-6 weeks and we don't want to have to decide between providing a timely, important security update to our users and breaking the browser extension (and therefore webapps)."

This is a very valid concern, and we are 100% aware of that. We tried to make sure that the QA infrastructure is ready for that,

[Jamie Strandboge (jdstrand)]

Thanks for your response, this is very encouraging because it is imperative that the security updates go out when they are available.

I think we need to define a process for coordinating your testing with the security/Desktop team(s). This will probably start with you guys tracking the beta channel early and culminate with some sort of notification to your team via a bug for the security update that introduces the new browser. Can you work with Micah (getting Chris Coulson's input would be good too, since there may be non-security SRUs and he is already dealing with the global menu) on defining this process? This process will be required for main inclusion.

[Víctor R. Ruiz (vrruiz)]

Jamie:

- There is a document for the manual tests. The document specify precise steps to check that the browser extensions are installed and work correctly, along all the desktop integration features of Web Apps, both for Firefox and Chromium.

- Firefox has automated tests for the extension, using Mozmill.

- Although not everything can be checked automatically right now due to introspection limitations, we have developed automated tests using Autopilot and Selenium Web Driver for both Chromium and Firefox.

- We have continuous integration for Web Apps project, which run unit tests. Currently, manual/functional tests are run by me (Web Apps QA Engineer) on an up-to-date virtual machine weekly, but work is in progress to run automated functional tests in Jenkins.

[Alexandre Abreu (abreu-alexandre)]

Jamie:

on the chromium side we are already involved through Ken VanDine with the chromium team, and working along with the releases and working off the beta channels before they reach stable. The process is taking shape and we have tried to be responsive so far,

We will most likely work to put a proper workflow in place to make the process as smooth as possible in this regard (security updates, stable releases, etc.).

[Jamie Strandboge (jdstrand)]

Glad to hear about the tests-- sounds like you will uncover the problems sooner than later before we need to perform a security update, which is great.

I want to be clear though-- we are most likely going to have a 0-day update for firefox when 12.10 is released so a plan for interfacing with my team regarding testing is a requirement for main inclusion. Otherwise, my team will be forced to publish a browser update without webapps verification from your team.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libunity-webapps - 2.3.3-0quantal1

---------------
libunity-webapps (2.3.3-0quantal1) quantal-proposed; urgency=low

  * New upstream release.
    - Don't add all webapps to the messaging menu (LP: #1052568)
    - Update WebApps to support dpkg distribution mechanism (LP: #1043461)
  * debian/unity-webapps-service.install
    - install all of /usr/bin
  * debian/control
    - dropped build depends for libindicate and added build dep for
      libmessaging-menu-dev
  * debian/libunity-webapps0.symbols
    - refreshed
 -- Ken VanDine <email address hidden> Wed, 19 Sep 2012 09:36:25 -0400

[Chris Coulson (chrisccoulson)]

And following on from Jamie's comment about a 0-day update for Firefox when 12.10 is released - at this stage of the 6-week release cycle (for Firefox), all addons should be compatible with the next release already. I expect to have functioning, compatible addons to be available at the start of every beta cycle so that we can offer them to our beta audience (https://launchpad.net/~mozillateam/+archive/firefox-next). We already ensure this is the case for our other addons.

For Firefox 16, this means that compatible addons should have been available by the end of August, and any changes necessary to make addons work with Firefox 17 should be complete by 9th October (with only critical bug-fixes during the 6-week beta cycle after that). This should be no problem though, seeing as you're all tracking nightly builds of Firefox, right?? ;) (and if not, this is pretty much mandatory for addon developers).

[Jamie Strandboge (jdstrand)]

Can we someone from the webapps team to commit to following Chris' advice and to track nightly builds so problems are found early and addressed way before the browser updates hit the archive?

[Jamie Strandboge (jdstrand)]

Also, could someone send an email to security <at> ubuntu.com CC'ing Chris with the plan regarding testing?

[Víctor R. Ruiz (vrruiz)]

Addons compatibility has already been tested with Firefox 16 and 17, using Mozilla's tarballs. We'll continue to be testing them along the cycle.

[Jamie Strandboge (jdstrand)]

I'm sorry, I wasn't clear. I know we are ready now, my concern is future testing and support in the stable releases for the full duration of their support.

[Víctor R. Ruiz (vrruiz)]

To be more clear. Currently, our developers are testing the addons in the current and next versions of Firefox (16, 17), using both nightly builds PPA and tarballs directly from Mozilla. Currently, we are working in the infrastructure to automate that tests in Jenkins. So we have manual tests, done periodically, and plan to have automated tests for the supported platforms. This testing will continue for the duration of the support period for stable Ubuntu releases.

[Olli Ries (ories)]

@comment 23: the described process will have to be applied to all supported/maintained versions of FF going forward. My understanding is that we are covered for now, but adding more versions is a concern, which to me should be addressed by the test automation vrruiz is working on.

[Jamie Strandboge (jdstrand)]

The conversation has started surrounding processes for liaising with the security, so no need to block on this any more.

ACK from security.