hook Oxide into Ubuntu platform API for media-hub

[Jamie Strandboge]

Right now oxide uses software rendering for audio and video via the chromium content api ffmpeg implementation for libffmpegsumo.so. This is provided in either oxideqt-codecs (suitable for main) or oxideqt-codecs-extra (suitable for universe). oxideqt-codecs-extra includes mp3 and h264 playback. Software rendering is not ideal on the phone, but in addition to that, webbrowser-app, webapp-container and apps using Oxide or UbuntuWebView are subject to application lifecycle and therefore the user experience for things such as grooveshark playlists and youtube videos is poor because the device will shut off the screen or suspend during playback.

The correct way to solve this is for Oxide to use the media-hub in some manner. This was somewhat easily solved with QtWebKit since it used gstreamer and it could be made to hook into the media-hub. However, QtWebKit is dead upstream which is one reason why we are using Oxide. We should look to see how qtwebengine is handling this (ie, are they just using the software rendering or are they redoing the QtWebKit gstreamer work for qtwebengine). Perhaps we could provide our own libffmpegsumo.so or wrap it in some manner. Perhaps we can work with upstream to have something that works better on Linux/Ubuntu in general. Whatever method is chosen should be maintainable in the face of weekly or biweekly stable updates (low api churn) and 6-8 week beta updates (potential api churn) to the chromium content api, since we expect this update frequency in our stable releases for security, bug and web compatibility fixes over a period of up to 5 years.

[Bill Filler (bfiller)]

when media-hub is fully in place there will be a platform API to provide media playback control and textures that will need to be integrated with Oxide for video playback

[Pat McGowan (pat-mcgowan)]

I see that Chromium has an abstraction of sorts (webmediaplayer_impl). I notice that a couple years back the Meego tv browser implemented this using their Unified MultiMedia Service (UMMS) which utilized gstreamer. None of that implies it would be easy.

[Chris Coulson (chrisccoulson)]

Ah, thanks. The abstraction is actually blink::WebMediaPlayer, and webmediaplayer_impl is just one implementation of that. But there is also an Android implementation too.

Is that Meego code available anywhere?

[Pat McGowan (pat-mcgowan)]

Its at <email address hidden>:meego-middleware/meego-tv-browser.git

[Rick Spencer (rick-rickspencer3)]

Setting to High importance, as this is not an RTM blocker.

[Noemí (noemi-gallego)]

I supose that this bug will fix problems when reproducing a video on browser and receiving a call, making video go on pause. Are there any progresses?

[Pat McGowan (pat-mcgowan)]

The initial support will be for audio only, video will come later. The webapps will need to explicitly opt in to this functionality so it will be deployed app by app once the oxide and mh support lands.

[Noemí (noemi-gallego)]

I've checked it on the new R8 version and streaming video doesn't pause while the device is ringing but it does when you hang up the call.

[David Barth (dbarth)]

The branch which landed in Oxide trunk is only for audio support, and will be enabled on a per web application basis. However, this is a significant milestone in connecting Oxide and the Media Hub. Video support is being developed now, and is based on that initial integration work.

[John McAleely (john.mcaleely)]

@dbarth - will video land on the ww05-2015 milestone or later?

[David Barth (dbarth)]

Later, we're talking about many weeks of work before this can perform with
the stability and performance level required for switching.

On Mon, Jan 19, 2015 at 3:28 PM, John McAleely <email address hidden>
wrote:

> @dbarth - will video land on the ww05-2015 milestone or later?
>
> --
> You received this bug notification because you are a member of Oxide
> Developers, which is subscribed to oxide-qt in Ubuntu.
> Matching subscriptions: oxide in Ubuntu
> https://bugs.launchpad.net/bugs/1249387
>
> Title:
> hook Oxide into Ubuntu platform API for media-hub
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/canonical-devices-system-image/+bug/1249387/+subscriptions
>

[John McAleely (john.mcaleely)]

Understood - thanks for the details!

On 20 January 2015 at 09:46, David Barth <email address hidden> wrote:

> Later, we're talking about many weeks of work before this can perform with
> the stability and performance level required for switching.
>
> On Mon, Jan 19, 2015 at 3:28 PM, John McAleely <<email address hidden>
> >
> wrote:
>
> > @dbarth - will video land on the ww05-2015 milestone or later?
> >
> > --
> > You received this bug notification because you are a member of Oxide
> > Developers, which is subscribed to oxide-qt in Ubuntu.
> > Matching subscriptions: oxide in Ubuntu
> > https://bugs.launchpad.net/bugs/1249387
> >
> > Title:
> > hook Oxide into Ubuntu platform API for media-hub
> >
> > To manage notifications about this bug go to:
> >
> >
> https://bugs.launchpad.net/canonical-devices-system-image/+bug/1249387/+subscriptions
> >
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1403478).
> https://bugs.launchpad.net/bugs/1249387
>
> Title:
> hook Oxide into Ubuntu platform API for media-hub
>
> Status in the base for Ubuntu mobile products:
> Fix Committed
> Status in Media Hub:
> Fix Committed
> Status in Oxide Webview:
> Fix Committed
> Status in oxide-qt package in Ubuntu:
> Fix Committed
>
> Bug description:
> Right now oxide uses software rendering for audio and video via the
> chromium content api ffmpeg implementation for libffmpegsumo.so. This
> is provided in either oxideqt-codecs (suitable for main) or oxideqt-
> codecs-extra (suitable for universe). oxideqt-codecs-extra includes
> mp3 and h264 playback. Software rendering is not ideal on the phone,
> but in addition to that, webbrowser-app, webapp-container and apps
> using Oxide or UbuntuWebView are subject to application lifecycle and
> therefore the user experience for things such as grooveshark playlists
> and youtube videos is poor because the device will shut off the screen
> or suspend during playback.
>
> The correct way to solve this is for Oxide to use the media-hub in
> some manner. This was somewhat easily solved with QtWebKit since it
> used gstreamer and it could be made to hook into the media-hub.
> However, QtWebKit is dead upstream which is one reason why we are
> using Oxide. We should look to see how qtwebengine is handling this
> (ie, are they just using the software rendering or are they redoing
> the QtWebKit gstreamer work for qtwebengine). Perhaps we could provide
> our own libffmpegsumo.so or wrap it in some manner. Perhaps we can
> work with upstream to have something that works better on Linux/Ubuntu
> in general. Whatever method is chosen should be maintainable in the
> face of weekly or biweekly stable updates (low api churn) and 6-8 week
> beta updates (potential api churn) to the chromium content api, since
> we expect this update frequency in our stable releases for security,
> bug and web compatibility fixes over a period of up to 5 years.
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/canonical-devices-system-image/+bug/1249387/+subscriptions
>

[Mitchell (curious-mitchell)]

So great to see this finally happening! Will edit the code for several of my webapps when it becomes available - where will it be anounced. Cheers.

[Launchpad Janitor (janitor)]

This bug was fixed in the package oxide-qt - 1.5.3-0ubuntu2

---------------
oxide-qt (1.5.3-0ubuntu2) vivid; urgency=medium

  * Update to v1.5.3
    - Bump Chromium rev to 41.0.2272.53
    - Fix LP: #1402975 - set a minimum viewport width of 980px on mobile
      for sites that don't specify a viewport width. Fixes layout of some
      desktop sites, and matches the behaviour of Chrome on Android
    - Fix LP: #1412981 - Geolocation permission requests from subframes are
      not cancelled if the frame is deleted or navigated to another site
    - Fix LP: #1379776 - Spoof Android for the purposes of GPU feature
      blacklisting on the phone. This means accelerated 2d canvas is now
      enabled on all devices that Chrome enables it on (including the Nexus 4),
      and other GPU features (eg, GPU rasterization) are enabled on devices
      where Chrome supports it
    - Fix LP: #1370366 - Add LocationBarController API, which allows the
      renderer compositor to calculate the position of the browser top header.
      This enables the header to be animated in sync with the content on
      scrolling
    - Fix LP: #1408136 - Committed LoadEvents fire for subframe loads when
      they shouldn't
    - Fix LP: #1377198 - CertificateError is not cancelled if you stop the
      pending navigation
    - Fix LP: #1373383 - The video player on Youtube does not resize correctly
      when transitioning from landscape to portrait. Ensure we correctly
      update the screen geometry and orientation visible to web content when
      the device orientation changes, and turn on orientation events
    - Really fix LP: #1337506 - Abort with
      "FATAL:texture_manager.cc(76)] Check failed: texture_count_ == 0u (1 vs. 0)"
      on shutdown
    - Fix LP: #1398044 - Fails to build with Qt 5.4
    - Fix LP: #1249387 - Add experimental support for playing audio through
      mediahub with the HTML media elements (Off by default. Video and
      MediaSource are not yet implemented. Not built yet because it depends
      on packages in universe)
    - Fix LP: #1417042 - Remove inactive touch points from the current touch
      state when we process a new touch event
    - Fix LP: #1417963 - Adapt to behaviour change in handling of QVariants
      from QML to C++ in Qt 5.4
    - Update the form factor detection code to not rely on the EGL vendor for
      detecting that we're on a device - use the Android system properties
      instead
    - Tidy up WebContents ownership during unload by adding a new singleton
      responsible for handling unloading (WebContentsUnloader)
    - Refactor WebFrame to be based around RenderFrameHost rather than
      FrameTreeNode, which we aren't meant to be using
    - Call QDesktopServices::openUrl on the UI thread. Failures are no longer
      propagated back to the resource dispatcher, so all attempts to open a
      URL externally will result in the load being cancelled whether there
      is an external handler or not
  * Update debian/control to add extra build dependencies:
    - libandroid-properties-dev on armhf, used for gathering data for GPU
      feature blacklisting when running on the phone
  * Update debian/l...

[David Barth (dbarth)]

Updating tasks statuses: MH integration is in Oxide trunk, but had to be temporarily reverted in the actual package / release. The new MH packaging branch promises to let Oxide, with MH support to land back in main, with proper dependencies to allow for that.

[Launchpad Janitor (janitor)]

This bug was fixed in the package oxide-qt - 1.5.5-0ubuntu0.14.10.2

---------------
oxide-qt (1.5.5-0ubuntu0.14.10.2) utopic-security; urgency=medium

  * Update to v1.5.5 - see USN-2521-1
    - Bump Chromium rev to 41.0.2272.76
    - Fix LP: #1427828 - Crash in WebFrame.sendMessageNoReply
    - Fix LP: #1411159 - Add work-around for shutdown crash in
      content::GpuChannelHost::Send, by deliberately leaking the
      cc::ContextProvider referenced by any task posted to the UI thread if
      the task never got a chance to run and is destroyed when the main loop
      is torn down. Note that 1.6 has the longer term fix for this crash.
    - Fix LP: #1421339 - Two scrollbars shown on the phone at a page scale of 1.
      + This is fixed by disabling the outer viewport scrollbars when
        pinch-viewport is on (copying Chrome on Android).
      + Also style the pinch viewport overlay scrollbars like Chrome on Android,
        as opposed to using the default desktop style that are designed to be
        dragged with a pointer
    - Fix LP: #1423531 - LoadEvent.isError is false for the error page if the
      navigation is triggered from a session restore or reload
    - Fix LP: #1402975 - set a minimum viewport width of 980px on mobile
      for sites that don't specify a viewport width. Fixes layout of some
      desktop sites, and matches the behaviour of Chrome on Android
    - Fix LP: #1412981 - Geolocation permission requests from subframes are
      not cancelled if the frame is deleted or navigated to another site
    - Fix LP: #1379776 - Spoof Android for the purposes of GPU feature
      blacklisting on the phone. This means accelerated 2d canvas is now
      enabled on all devices that Chrome enables it on (including the Nexus 4),
      and other GPU features (eg, GPU rasterization) are enabled on devices
      where Chrome supports it
    - Fix LP: #1370366 - Add LocationBarController API, which allows the
      renderer compositor to calculate the position of the browser top header.
      This enables the header to be animated in sync with the content on
      scrolling
    - Fix LP: #1408136 - Committed LoadEvents fire for subframe loads when
      they shouldn't
    - Fix LP: #1377198 - CertificateError is not cancelled if you stop the
      pending navigation
    - Fix LP: #1373383 - The video player on Youtube does not resize correctly
      when transitioning from landscape to portrait. Ensure we correctly
      update the screen geometry and orientation visible to web content when
      the device orientation changes, and turn on orientation events
    - Really fix LP: #1337506 - Abort with
      "FATAL:texture_manager.cc(76)] Check failed: texture_count_ == 0u (1 vs. 0)"
      on shutdown
    - Fix LP: #1398044 - Fails to build with Qt 5.4
    - Fix LP: #1249387 - Add experimental support for playing audio through
      mediahub with the HTML media elements (Off by default. Video and
      MediaSource are not yet implemented. Not built yet because it depends
      on packages in universe)
    - Fix LP: #1417042 - Remove inactive touch points from the current touch
      state when we process a new touch event
    - Fix ...

[Launchpad Janitor (janitor)]

This bug was fixed in the package oxide-qt - 1.5.5-0ubuntu0.14.04.3

---------------
oxide-qt (1.5.5-0ubuntu0.14.04.3) trusty-security; urgency=medium

  * Update to v1.5.5 - see USN-2521-1
    - Bump Chromium rev to 41.0.2272.76
    - Fix LP: #1427828 - Crash in WebFrame.sendMessageNoReply
    - Fix LP: #1411159 - Add work-around for shutdown crash in
      content::GpuChannelHost::Send, by deliberately leaking the
      cc::ContextProvider referenced by any task posted to the UI thread if
      the task never got a chance to run and is destroyed when the main loop
      is torn down. Note that 1.6 has the longer term fix for this crash.
    - Fix LP: #1421339 - Two scrollbars shown on the phone at a page scale of 1.
      + This is fixed by disabling the outer viewport scrollbars when
        pinch-viewport is on (copying Chrome on Android).
      + Also style the pinch viewport overlay scrollbars like Chrome on Android,
        as opposed to using the default desktop style that are designed to be
        dragged with a pointer
    - Fix LP: #1423531 - LoadEvent.isError is false for the error page if the
      navigation is triggered from a session restore or reload
    - Fix LP: #1402975 - set a minimum viewport width of 980px on mobile
      for sites that don't specify a viewport width. Fixes layout of some
      desktop sites, and matches the behaviour of Chrome on Android
    - Fix LP: #1412981 - Geolocation permission requests from subframes are
      not cancelled if the frame is deleted or navigated to another site
    - Fix LP: #1379776 - Spoof Android for the purposes of GPU feature
      blacklisting on the phone. This means accelerated 2d canvas is now
      enabled on all devices that Chrome enables it on (including the Nexus 4),
      and other GPU features (eg, GPU rasterization) are enabled on devices
      where Chrome supports it
    - Fix LP: #1370366 - Add LocationBarController API, which allows the
      renderer compositor to calculate the position of the browser top header.
      This enables the header to be animated in sync with the content on
      scrolling
    - Fix LP: #1408136 - Committed LoadEvents fire for subframe loads when
      they shouldn't
    - Fix LP: #1377198 - CertificateError is not cancelled if you stop the
      pending navigation
    - Fix LP: #1373383 - The video player on Youtube does not resize correctly
      when transitioning from landscape to portrait. Ensure we correctly
      update the screen geometry and orientation visible to web content when
      the device orientation changes, and turn on orientation events
    - Really fix LP: #1337506 - Abort with
      "FATAL:texture_manager.cc(76)] Check failed: texture_count_ == 0u (1 vs. 0)"
      on shutdown
    - Fix LP: #1398044 - Fails to build with Qt 5.4
    - Fix LP: #1249387 - Add experimental support for playing audio through
      mediahub with the HTML media elements (Off by default. Video and
      MediaSource are not yet implemented. Not built yet because it depends
      on packages in universe)
    - Fix LP: #1417042 - Remove inactive touch points from the current touch
      state when we process a new touch event
    - Fix ...

[Mitchell (curious-mitchell)]

Hi, am running vivid-proposed on my nexus 7 - any idea if this fix has hit the images yet? I notice my ABC Radio webapp still falls silent when focus is lost, or the screen turns off. Do I need to update the webapp-container I'm using in order to see this change?

Thanks.

M

[David Barth (dbarth)]

The webapp-container part is landing. You need to add this flag to your
desktop file:

--enable-media-hub-audio

However, we still need to wait for the next release of Oxide (1.7) to have
the integration enabled again. This is planned for next week, in the
6-weeks cadence of the Oxide releases.

On Sun, Apr 12, 2015 at 2:08 PM, Mitchell <email address hidden> wrote:

> Hi, am running vivid-proposed on my nexus 7 - any idea if this fix has
> hit the images yet? I notice my ABC Radio webapp still falls silent when
> focus is lost, or the screen turns off. Do I need to update the webapp-
> container I'm using in order to see this change?
>
> Thanks.
>
> M
>
> --
> You received this bug notification because you are a bug assignee.
> Matching subscriptions: oxide in Ubuntu
> https://bugs.launchpad.net/bugs/1249387
>
> Title:
> hook Oxide into Ubuntu platform API for media-hub
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/canonical-devices-system-image/+bug/1249387/+subscriptions
>

[Mitchell (curious-mitchell)]

Thanks, looking forward to it.

M

On 13/04/15 18:37, David Barth wrote:
> The webapp-container part is landing. You need to add this flag to your
> desktop file:
>
> --enable-media-hub-audio
>
> However, we still need to wait for the next release of Oxide (1.7) to have
> the integration enabled again. This is planned for next week, in the
> 6-weeks cadence of the Oxide releases.
>
> On Sun, Apr 12, 2015 at 2:08 PM, Mitchell <email address hidden>
> wrote:
>
>> Hi, am running vivid-proposed on my nexus 7 - any idea if this fix has
>> hit the images yet? I notice my ABC Radio webapp still falls silent when
>> focus is lost, or the screen turns off. Do I need to update the webapp-
>> container I'm using in order to see this change?
>>
>> Thanks.
>>
>> M
>>
>> --
>> You received this bug notification because you are a bug assignee.
>> Matching subscriptions: oxide in Ubuntu
>> https://bugs.launchpad.net/bugs/1249387
>>
>> Title:
>> hook Oxide into Ubuntu platform API for media-hub
>>
>> To manage notifications about this bug go to:
>>
>> https://bugs.launchpad.net/canonical-devices-system-image/+bug/1249387/+subscriptions
>>

[Mitchell (curious-mitchell)]

Hi David, just wondering if this has landed yet? I'm on vivid-proposed with my Nexus 7 (image 194), and building with the above flag doesn't work. When using this flag in the desktop file:
--enable-media-hub-audio
I get no audio whatsoever. Is there anything else I need to add or do? I'm using ubuntu-sdk-15.04

Thanks.

[David Barth (dbarth)]

There is a regression in Oxide, which recent builds have not been linked against media-hub again :/ Still Investigating why.

Your flag is correct, and the version of webapp-container shipping in vivid is correct as well. You can verify that by grepping for OXIDE in the /proc/<pid>/environ of the oxide-sandbox process associated with your webapp instance.

[David Barth (dbarth)]

See https://bugs.launchpad.net/oxide/+bug/1460705

[Justin McPherson (justinmcp)]

There has been some off report discussion on this issue, resolving around expectations for background playback.

Status - currently no persistent background playback, awaiting resolution of #1425818.

[David Barth (dbarth)]

I have created this bug to document the work done on audio streams background playback: https://bugs.launchpad.net/oxide/+bug/1471786

As a result the fuller MH integration can be reopened here to clarify the general integration status.

[David Barth (dbarth)]

Media arbitration is the next big step in that integration. The goal is not to render media with media-hub directly, but to ensure that media rendering performed by the oxide stack is done in coordination with the central media-hub service.

See attached branch: https://code.launchpad.net/~justinmcp/oxide/media-arbitration