BBC changed protocol, causing reduced functionality

[linuxcentre]

Binary package hint: get-iplayer

Issues caused by the BBC changing protocol:
* iPhone mode no longer works as the BBC changed protocol.
* wma radio mode doesn't work since BBC changed it.
* Changes from BBC require the retry attempt limit to workaround more frequent retries of flash streams.
* several flash modes not working due to bugs found in CDN metadata parser.
* Flashaac radio modes do not working as expected since the BBC added multiple flash aac bitrates to all radio programmes.
* BBC local radio stations listing changed from BBC, package should be updated to reflect this.

This issues are fixed in the current development version of Ubuntu.

This package has no reverse depends, so the regression potential is minor. The branch attached is based on a patch provided by upstream, based on them backporting the fix to the bug.

TEST CASE: dave@micmini:~/tmp$ get-iplayer -g 860 --modes=iphone
....
ERROR: Failed to record ...

[Jonathan Wiltshire (jwiltshire)]

Can you pin down a minimal patch for it against the Karmic version, and we can get it in to a stable update?

Thanks

[linuxcentre (linuxcentre)]

On Fri, 2010-01-15 at 15:25 +0000, Jonathan Wiltshire wrote:
> Can you pin down a minimal patch for it against the Karmic version, and
> we can get it in to a stable update?
>
> Thanks

Hi Jonathan,

I had a go at this but in the 10 releases between 2.41 and 2.51 quite a
bit of this code was changed such that I am not making much useful
progress without breaking other functionality. So I'm going to have to
say 'no' in this case to save me many hours work changing and testing.

BTW: Why does Karmic not just follow upstream Debian? Is there some
policy surrounding this process?

Best Regards

Phil

[Jonathan Wiltshire (jwiltshire)]

On Fri, Jan 15, 2010 at 06:59:26PM -0000, linuxcentre wrote:
> I had a go at this but in the 10 releases between 2.41 and 2.51 quite a
> bit of this code was changed such that I am not making much useful
> progress without breaking other functionality. So I'm going to have to
> say 'no' in this case to save me many hours work changing and testing.

Hmm. A whole new upstream won't be accepted for Karmic through the updates
mechanism, it's too big a change for a stable release. I'm afraid my
Perl-fu isn't really strong enough.

> BTW: Why does Karmic not just follow upstream Debian? Is there some
> policy surrounding this process?

Because Karmic is a stable release, so it must be touched only as little as
possible. The get_iplayer package in Debian hasn't ever been in a release,
so we haven't had that situation yet. When Debian 6 "Squeeze" is released
the process will be similar; in Ubuntu Lucid the automatic sync from
Squeeze is still taking place. That's why there are two versions showing
up.

>
> Best Regards
>
> Phil
>

--
Jonathan Wiltshire

1024D: 0xDB800B52 / 4216 F01F DCA9 21AC F3D3 A903 CA6B EA3E DB80 0B52
4096R: 0xD3524C51 / 0A55 B7C5 1223 3942 86EC 74C3 5394 479D D352 4C51

[linuxcentre (linuxcentre)]

Hi,

Thanks for explaining that.

I've done a fuller analysis of the differences and bugs in the current
Ubuntu version. There are also several BBC introduced bugs that are
fixed in subsequent versions of get_iplayer. They are:

* AAC mode changes by BBC: flash AAC recordings would end up unavailable
or very low quality - fixed in v2.43.

* BBC Live TV mode changes: Users would only get the first mode of 3
modes available for some channels. - fixed in v2.43.

* Fixed mms and rtsp url parsing from asx and ram playlists which now
parses new wma radio stream metadata. The BBC are gradually switching
off RealAudio and using WMA instead so this becomes ever more important
a fix (especially for international users). Fixed in v2.50

* iPhone mode breakage fixed (as per bug report). - fixed in v2.51

Also, a get_iplayer bug which caused HD to fail 50% of times for some:

* bug with authstring typo in level3/iplayertok CDN rtmp parameters
which caused iPlayer HD to often fail. Fixed in v2.45

Finally, I'd strongly recommend using v2.53 since that also fixes the
problems where /var/lib/get_iplayer/ was used for system-wide options.
This version reverts to using /etc/get_iplayer/. At very least go with
version v2.51 and add your usual patch to use /etc/get_iplayer/.

Very little extra functionality went in between 2.41 and 2.53 and
certainly nothing which has since broken.

Best Regards

Phil

On Fri, 2010-01-15 at 22:56 +0000, Jonathan Wiltshire wrote:
> On Fri, Jan 15, 2010 at 06:59:26PM -0000, linuxcentre wrote:
> > I had a go at this but in the 10 releases between 2.41 and 2.51 quite a
> > bit of this code was changed such that I am not making much useful
> > progress without breaking other functionality. So I'm going to have to
> > say 'no' in this case to save me many hours work changing and testing.
>
> Hmm. A whole new upstream won't be accepted for Karmic through the updates
> mechanism, it's too big a change for a stable release. I'm afraid my
> Perl-fu isn't really strong enough.
>
> > BTW: Why does Karmic not just follow upstream Debian? Is there some
> > policy surrounding this process?
>
> Because Karmic is a stable release, so it must be touched only as little as
> possible. The get_iplayer package in Debian hasn't ever been in a release,
> so we haven't had that situation yet. When Debian 6 "Squeeze" is released
> the process will be similar; in Ubuntu Lucid the automatic sync from
> Squeeze is still taking place. That's why there are two versions showing
> up.
>
> >
> > Best Regards
> >
> > Phil
> >
>
> --
> Jonathan Wiltshire
>
> 1024D: 0xDB800B52 / 4216 F01F DCA9 21AC F3D3 A903 CA6B EA3E DB80 0B52
> 4096R: 0xD3524C51 / 0A55 B7C5 1223 3942 86EC 74C3 5394 479D D352 4C51
>

[linuxcentre (linuxcentre)]

Hi,
Since the BBC changed the iPhone streaming method yet again last week (now it's similar to 2.41 again!) I have created an absoluetly minimal patch for all the critical show-stopper (!) bugs. I've also fixed other things that were either due to BBC changing their stuff which broke get_iplayer recordings which have been fixed in trunk and backported.

Here is a summary of the bugfix changes:

* Fixed iPhone mode since BBC changed it.
* Fixed wma radio mode since BBC changed it.
* Increased retry attempt limit to workaround more frequent retries of flash streams since BBC made changes.
* Fixed existing bugs found in CDN metadata parsers that prevented several flash modes working.
* Fixed flashaac radio modes since the BBC added multiple flash aac bitrates to all radio programmes.
* Added a few missing BBC local radio stations.

I've attached the diff against get_iplayer-2.41 below.

[John Dong (jdong)]

Hi Dave and linuxcentre,

Thanks for filing this SRU request. A few comments:

(1) The patch's fixes, as detailed in debian/changelog and linuxcentre's comments #4 and #5 do not match this bug's original description. These two should be made consistent for SRU verification purposes. Reading over the fixes, I agree that ALL of these deserve being in the SRU, so I suggest just updating the bug description with all the bugs being fixed.

(2) Minor quibbles: No need to mention MaintainerSpec updating in the SRU changelog. Users pulling this update probably don't care.

(3) karmic --> karmic-proposed in changelog

(4) "Closes (LP: #12345)" is nonstandard. Just "(LP: #12345)" for closing Launchpad bugs, and "Closes: #12345" for Debian bugs.

(5) Version number should be 2.41-1ubuntu0.1

Please address these issues, and I'd be happy to look at this SRU again!

Thanks,

John

[John Dong (jdong)]

as of revision 8 on the bzr branch, everything looks good. ACK from ubuntu-sru and continue with upload!

[Rich Johnson (nixternal)]

Just uploaded Dave's changes, sitting in "Waiting for approval" now.

[Jonathan Riddell (jr)]

accepted into karmic-proposed, please test

[Dave Walker (davewalker)]

I can confirm that the -proposed package closes this bug. Would be quite nice if someone else could resolve the verification tag, especially as it was my upload.

[Martin Pitt (pitti)]

This also still needs to be fixed in lucid _first_ before it can go to -updates.

[Dave Walker (davewalker)]

@Martin, It would be really nice if bug #557651 could be sync'd, that seems to make the best sense long term; and mean that this bug is fixed in Lucid.

[Launchpad Janitor (janitor)]

This bug was fixed in the package get-iplayer - 2.78-1

---------------
get-iplayer (2.78-1) unstable; urgency=low

  * New upstream release
      Closes: #582812, #582905
      LP: #505433
  * Update watch file
  * Patch limelight-cdn-582659 is integrated upstream
  * New patch manpage to include manual, temporarily until this is fixed
    upstream
  * Remove old NEWS
 -- Ubuntu Archive Auto-Sync <email address hidden> Mon, 31 May 2010 10:30:25 +0100

[Jean-Baptiste Lallement (jibel)]

SRU Verification for Karmic:

The version of get-iplayer 2.41-1ubuntu0.1 in -proposed only fix the radio streams but TV streams are still not working. The problem is the same in Maverick. This is probably due to the BBC's policy regarding open-source applications accessing iPlayer streams (http://linuxcentre.net/get_iplayer-dropped-in-response-to-bbcs-lack-of-support-for-open-source) Because of this and sadly, because linuxcentre ceased development of get-iplayer, it is unlikely that it will be fixed sometimes soon.

I'm marking as verification-failed and setting Karmic status to Won't fix.
Don't hesitate to reopen if BBC changes its mind.

[Jonathan Wiltshire (jwiltshire)]

On Wed, Jun 23, 2010 at 02:05:29PM -0000, Jean-Baptiste Lallement wrote:
> The version of get-iplayer 2.41-1ubuntu0.1 in -proposed only fix the
> radio streams but TV streams are still not working. The problem is the
> same in Maverick. This is probably due to the BBC's policy regarding
> open-source applications accessing iPlayer streams
> (http://linuxcentre.net/get_iplayer-dropped-in-response-to-bbcs-lack-of-
> support-for-open-source) Because of this and sadly, because linuxcentre
> ceased development of get-iplayer, it is unlikely that it will be fixed
> sometimes soon.
>
> I'm marking as verification-failed and setting Karmic status to Won't fix.
> Don't hesitate to reopen if BBC changes its mind.

Phil Lewis may not be developing get_iplayer any more, but it is continuing
at http://git.infradead.org/get_iplayer.git.

TV streaming was fixed in this tree, at least until the last time I used it
the other day.

--
Jonathan Wiltshire

4096R: 0xD3524C51 / 0A55 B7C5 1223 3942 86EC 74C3 5394 479D D352 4C51

[Jean-Baptiste Lallement (jibel)]

Thanks for the info. That's good news that get_iplayer is still alive. But that is still a verification-failed for karmic unless the tv streaming fix is backported to Karmic.