Ubuntu
get-flash-videos package

get-flash-videos failing to download some BBC news videos (possibly other BBC videos too)

Bug #1210314 reported by Pigeon
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
get-flash-videos (Ubuntu)
New
Undecided
Unassigned

Bug Description

System: Ubuntu 11.04 (but bug is not related to any other packages in system)
Package: get-flash-videos 1.25~git2012.06.27-1

get-flash-videos is failing on certain videos from the BBC, apparently because of a change in the BBC web page code.
Example output from failed run:

$ get_flash_videos --debug http://www.bbc.co.uk/news/uk-23626232
2 plugins installed:
- Hulu.pm
- Itv.pm
Trying to open plugin ~/.get_flash_videos/plugins/Www.pm
Trying to open plugin ~/.get_flash_videos/plugins/Bbc.pm
Using method 'bbc' for http://www.bbc.co.uk/news/uk-23626232
Downloading http://www.bbc.co.uk/news/uk-23626232
-> GET http://www.bbc.co.uk/news/uk-23626232
<- 200 text/html text/html; charset=utf-8 (58275)
Error: Couldn't find BBC XML playlist URL in http://www.bbc.co.uk/news/uk-23626232 at /usr/share/perl5/FlashVideo/Site/Bbc.pm line 60.

Couldn't extract Flash movie URL. This site may need specific support adding,
or fixing.

Please confirm the site is using Flash video and if you have Flash available
check that the URL really works(!).

Check for updates by running: /usr/bin/get_flash_videos --update

If the latest version does not support this please open a bug (or
contribute a patch!) at http://code.google.com/p/get-flash-videos/
make sure you include the output with --debug enabled.
Couldn't download any videos.

The following patch fixes this problem...

$ diff -u /usr/share/perl5/FlashVideo/Site/Bbc.pm.orig /usr/share/perl5/FlashVideo/Site/Bbc.pm
--- /usr/share/perl5/FlashVideo/Site/Bbc.pm.orig 2013-08-08 23:18:52.000000000 +0100
+++ /usr/share/perl5/FlashVideo/Site/Bbc.pm 2013-08-08 22:56:07.000000000 +0100
@@ -18,6 +18,10 @@
   if ($browser->content =~ /<param name="playlist" value="(http:.+?\.s?xml)"/) {
     $playlist_xml = $1;
   }
+ elsif ($browser->content =~ /"href":"(http:[^"]+?playlist\.s?xml)"/) {
+ $playlist_xml = $1;
+ $playlist_xml =~ s/\\//g;
+ }
   elsif($browser->content =~ /empDivReady\s*\(([^)]+)/) {
     my @params = split /,\s*/, $1;

Example output from the same URL with this patch applied:

$ get_flash_videos --debug http://www.bbc.co.uk/news/uk-23626232
2 plugins installed:
- Hulu.pm
- Itv.pm
Trying to open plugin ~/.get_flash_videos/plugins/Www.pm
Trying to open plugin ~/.get_flash_videos/plugins/Bbc.pm
Using method 'bbc' for http://www.bbc.co.uk/news/uk-23626232
Downloading http://www.bbc.co.uk/news/uk-23626232
-> GET http://www.bbc.co.uk/news/uk-23626232
<- 200 text/html text/html; charset=utf-8 (58275)
Playlist URL: http://playlists.bbc.co.uk/news/uk-23626232A/playlist.sxml
-> GET http://playlists.bbc.co.uk/news/uk-23626232A/playlist.sxml
<- 200 text/xml (4864)
-> GET http://open.live.bbc.co.uk/mediaselector/4/mtis/stream/p01dwkgh
<- 200 text/xml (5841)
FlashVideo::RTMPDownloader=HASH(0x9f22610)
Running rtmpdump --verbose --pageUrl 'http://www.bbc.co.uk/news/uk-23626232' --app 'ondemand?_fcs_vhost=cp45413.edgefcs.net&auth=daEdWclbWbVb4dTc4bmckdtdId9bFaIbzdC-bsbbn8-bWG-JpmEIoDoLEsGtxE&aifp=v001&slist=public/mps_h264_200/public/news/uk/1001000/1001225_h264_176k.mp4;public/mps_h264_400/public/news/uk/1001000/1001225_h264_512k.mp4;public/mps_h264_med/public/news/uk/1001000/1001225_h264_800k.mp4;public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4' --tcUrl 'rtmp://cp45413.edgefcs.net/ondemand?_fcs_vhost=cp45413.edgefcs.net&auth=daEdWclbWbVb4dTc4bmckdtdId9bFaIbzdC-bsbbn8-bWG-JpmEIoDoLEsGtxE&aifp=v001&slist=public/mps_h264_200/public/news/uk/1001000/1001225_h264_176k.mp4;public/mps_h264_400/public/news/uk/1001000/1001225_h264_512k.mp4;public/mps_h264_med/public/news/uk/1001000/1001225_h264_800k.mp4;public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4' --swfUrl 'http://news.bbc.co.uk/player/emp/2.11.7978_8433/9player.swf' --rtmp 'rtmp://cp45413.edgefcs.net/ondemand' --flv 'BBC_-_Co-op_supermarket_to_stop_selling_Nuts_magazine.flv' --playpath 'mp4:public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4?auth=daEdWclbWbVb4dTc4bmckdtdId9bFaIbzdC-bsbbn8-bWG-JpmEIoDoLEsGtxE&aifp=v001&slist=public/mps_h264_200/public/news/uk/1001000/1001225_h264_176k.mp4;public/mps_h264_400/public/news/uk/1001000/1001225_h264_512k.mp4;public/mps_h264_med/public/news/uk/1001000/1001225_h264_800k.mp4;public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4'
RTMPDump v2.4
(c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
rtmpdump: DEBUG: Parsing...
rtmpdump: DEBUG: Parsed protocol: 0
rtmpdump: DEBUG: Parsed host : cp45413.edgefcs.net
rtmpdump: DEBUG: Parsed app : ondemand
rtmpdump: DEBUG: Protocol : RTMP
rtmpdump: DEBUG: Hostname : cp45413.edgefcs.net
rtmpdump: DEBUG: Port : 1935
rtmpdump: DEBUG: Playpath : mp4:public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4?auth=daEdWclbWbVb4dTc4bmckdtdId9bFaIbzdC-bsbbn8-bWG-JpmEIoDoLEsGtxE&aifp=v001&slist=public/mps_h264_200/public/news/uk/1001000/1001225_h264_176k.mp4;public/mps_h264_400/public/news/uk/1001000/1001225_h264_512k.mp4;public/mps_h264_med/public/news/uk/1001000/1001225_h264_800k.mp4;public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4
rtmpdump: DEBUG: tcUrl : rtmp://cp45413.edgefcs.net/ondemand?_fcs_vhost=cp45413.edgefcs.net&auth=daEdWclbWbVb4dTc4bmckdtdId9bFaIbzdC-bsbbn8-bWG-JpmEIoDoLEsGtxE&aifp=v001&slist=public/mps_h264_200/public/news/uk/1001000/1001225_h264_176k.mp4;public/mps_h264_400/public/news/uk/1001000/1001225_h264_512k.mp4;public/mps_h264_med/public/news/uk/1001000/1001225_h264_800k.mp4;public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4
rtmpdump: DEBUG: swfUrl : http://news.bbc.co.uk/player/emp/2.11.7978_8433/9player.swf
rtmpdump: DEBUG: pageUrl : http://www.bbc.co.uk/news/uk-23626232
rtmpdump: DEBUG: app : ondemand?_fcs_vhost=cp45413.edgefcs.net&auth=daEdWclbWbVb4dTc4bmckdtdId9bFaIbzdC-bsbbn8-bWG-JpmEIoDoLEsGtxE&aifp=v001&slist=public/mps_h264_200/public/news/uk/1001000/1001225_h264_176k.mp4;public/mps_h264_400/public/news/uk/1001000/1001225_h264_512k.mp4;public/mps_h264_med/public/news/uk/1001000/1001225_h264_800k.mp4;public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4
rtmpdump: DEBUG: live : no
rtmpdump: DEBUG: timeout : 30 sec
rtmpdump: DEBUG: Setting buffer time to: 36000000ms
Connecting ...
rtmpdump: DEBUG: RTMP_Connect1, ... connected, handshaking
rtmpdump: DEBUG: HandShake: Type Answer : 03
rtmpdump: DEBUG: HandShake: Server Uptime : 739889075
rtmpdump: DEBUG: HandShake: Handshaking finished....
rtmpdump: DEBUG: HandleServerBW: server BW = 1250000
chunk size change to 4096
DEBUG: RTMP_ClientPacket, received: invoke 240 bytes
DEBUG: (object begin)
DEBUG: (object begin)
DEBUG: Property: <Name: fmsVer, STRING: FMS/4,5,2,524>
DEBUG: Property: <Name: capabilities, NUMBER: 127.00>
DEBUG: Property: <Name: mode, NUMBER: 1.00>
DEBUG: (object end)
DEBUG: (object begin)
DEBUG: Property: <Name: level, STRING: status>
DEBUG: Property: <Name: code, STRING: NetConnection.Connect.Success>
DEBUG: Property: <Name: description, STRING: Connection succeeded.>
DEBUG: Property: <Name: objectEncoding, NUMBER: 0.00>
DEBUG: Property: <Name: data, OBJECT>
DEBUG: (object begin)
DEBUG: Property: <Name: version, STRING: 4,5,2,524>
DEBUG: (object end)
DEBUG: (object end)
DEBUG: (object end)
DEBUG: HandleInvoke, server invoking <_result>
DEBUG: HandleInvoke, received result for method call <connect>
DEBUG: sending ctrl. type: 0x0003
DEBUG: Invoking createStream
DEBUG: RTMP_ClientPacket, received: invoke 21 bytes
DEBUG: (object begin)
DEBUG: Property: NULL
DEBUG: (object end)
DEBUG: HandleInvoke, server invoking <onBWDone>
DEBUG: Invoking _checkbw
rtmpdump: DEBUG: RTMP_ClientPacket, received: invoke 29 bytes
eInvoke, server invoking <_result>
DEBUG: HandleInvoke, received result for method call <createStream>
DEBUG: SendPlay, seekTime=0, stopTime=0, sending play: mp4:public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4?auth=daEdWclbWbVb4dTc4bmckdtdId9bFaIbzdC-bsbbn8-bWG-JpmEIoDoLEsGtxE&aifp=v001&slist=public/mps_h264_200/public/news/uk/1001000/1001225_h264_176k.mp4;public/mps_h264_400/public/news/uk/1001000/1001225_h264_512k.mp4;public/mps_h264_med/public/news/uk/1001000/1001225_h264_800k.mp4;public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4
DEBUG: Invoking play
DEBUG: sending ctrl. type: 0x0003
rtmpdump: DEBUG: RTMP_ClientPacket, received: invoke 10275 bytes
ndleInvoke, server invoking <_onbwcheck>
DEBUG: Invoking _result
rtmpdump: DEBUG: HandleCtrl, received ctrl. type: 4, len: 6
rtmpdump: DEBUG: HandleChangeChunkSize, received: chunk size change to 4096
eCtrl, Stream Begin 1
DEBUG: RTMP_ClientPacket, received: invoke 270 bytes
DEBUG: (object begin)
DEBUG: Property: NULL
DEBUG: (object begin)
DEBUG: Property: <Name: level, STRING: status>
DEBUG: Property: <Name: code, STRING: NetStream.Play.Reset>
DEBUG: Property: <Name: description, STRING: Playing and resetting public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4.>
DEBUG: Property: <Name: details, STRING: public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4>
DEBUG: Property: <Name: clientid, STRING: qAAOwpJA>
DEBUG: (object end)
DEBUG: (object end)
DEBUG: HandleInvoke, server invoking <onStatus>
DEBUG: HandleInvoke, onStatus: NetStream.Play.Reset
rtmpdump: DEBUG: RTMP_ClientPacket, received: invoke 264 bytes
operty: <Name: level, STRING: status>
DEBUG: Property: <Name: code, STRING: NetStream.Play.Start>
DEBUG: Property: <Name: description, STRING: Started playing public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4.>
DEBUG: Property: <Name: details, STRING: public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4>
DEBUG: Property: <Name: clientid, STRING: qAAOwpJA>
DEBUG: (object end)
DEBUG: (object end)
DEBUG: HandleInvoke, server invoking <onStatus>
DEBUG: HandleInvoke, onStatus: NetStream.Play.Start
Starting download at: 0.000 kB
DEBUG: RTMP_ClientPacket, received: notify 24 bytes
DEBUG: (object begin)
DEBUG: (object end)
DEBUG: ignoring too small audio packet: size: 0
DEBUG: RTMP_ClientPacket, received: notify 44 bytes
DEBUG: (object begin)
DEBUG: (object begin)
DEBUG: Property: <Name: code, STRING: NetStream.Data.Start>
DEBUG: (object end)
DEBUG: (object end)
rtmpdump: DEBUG: RTMP_ClientPacket, received: notify 494 bytes
   duration, NUMBER: 123.24>
DEBUG: Property: <Name: moovPosition, NUMBER: 36.00>
DEBUG: Property: <Name: width, NUMBER: 704.00>
DEBUG: Property: <Name: height, NUMBER: 396.00>
DEBUG: Property: <Name: videocodecid, STRING: avc1>
DEBUG: Property: <Name: audiocodecid, STRING: mp4a>
DEBUG: Property: <Name: avcprofile, NUMBER: 77.00>
DEBUG: Property: <Name: avclevel, NUMBER: 30.00>
DEBUG: Property: <Name: aacaot, NUMBER: 2.00>
DEBUG: Property: <Name: videoframerate, NUMBER: 25.00>
DEBUG: Property: <Name: audiosamplerate, NUMBER: 44100.00>
DEBUG: Property: <Name: audiochannels, NUMBER: 1.00>
DEBUG: Property: <Name: trackinfo, OBJECT>
DEBUG: (object begin)
DEBUG: (object begin)
DEBUG: Property: <Name: length, NUMBER: 3081000.00>
DEBUG: Property: <Name: timescale, NUMBER: 25000.00>
DEBUG: Property: <Name: language, STRING: und>
DEBUG: Property: <Name: sampledescription, OBJECT>
DEBUG: (object begin)
DEBUG: (object begin)
DEBUG: Property: <Name: sampletype, STRING: avc1>
DEBUG: (object end)
DEBUG: (object end)
rtmpdump: DEBUG: (object end)
: timescale, NUMBER: 44100.00>
DEBUG: Property: <Name: language, STRING: und>
DEBUG: Property: <Name: sampledescription, OBJECT>
DEBUG: (object begin)
DEBUG: (object begin)
DEBUG: Property: <Name: sampletype, STRING: mp4a>
DEBUG: (object end)
DEBUG: (object end)
DEBUG: (object end)
DEBUG: (object end)
DEBUG: (object end)
DEBUG: (object end)
INFO: Metadata:
INFO: duration 123.24
INFO: moovPosition 36.00
INFO: width 704.00
INFO: height 396.00
INFO: videocodecid avc1
INFO: audiocodecid mp4a
INFO: avcprofile 77.00
INFO: avclevel 30.00
INFO: aacaot 2.00
INFO: videoframerate 25.00
INFO: audiosamplerate 44100.00
INFO: audiochannels 1.00
INFO: trackinfo:
INFO: length 3081000.00
INFO: timescale 25000.00
INFO: language und
INFO: sampledescription:
INFO: sampletype avc1
INFO: length 5434368.00
INFO: timescale 44100.00
INFO: language und
INFO: sampledescription:
INFO: sampletype mp4a
DEBUG: RTMP_ClientPacket, received: invoke 57 bytesne.flv: 0% (53.28 / 53277.00 KiB)
DEBUG: (object begin)
DEBUG: Property: NULL
DEBUG: (object end)
DEBUG: HandleInvoke, server invoking <onBWDone>
rtmpdump: DEBUG: RTMP_ClientPacket, received: invoke 40 bytes
eInvoke, server invoking <_onbwdone>
BBC_-_Co-op_supermarket_to_stop_selling_Nuts_magazine.flv: 99% (22530.67 / 22712.37 KiB)rtmpdump: DEBUG: ignoring too small audio packet: size: 0
rtmpdump: DEBUG: RTMP_ClientPacket, received: notify 102 bytes
rtmpdump: DEBUG: (object begin)
rtmpdump: DEBUG: (object begin)
rtmpdump: DEBUG: Property: <Name: code, STRING: NetStream.Play.Complete>
rtmpdump: DEBUG: Property: <Name: level, STRING: status>
rtmpdump: DEBUG: Property: <Name: duration, NUMBER: 103.00>
rtmpdump: DEBUG: Property: <Name: bytes, NUMBER: 23271822.00>
rtmpdump: DEBUG: (object end)
rtmpdump: DEBUG: (object end)
rtmpdump: DEBUG: HandleCtrl, received ctrl. type: 1, len: 6
rtmpdump: DEBUG: HandleCtrl, Stream EOF 1
rtmpdump: DEBUG: RTMP_ClientPacket, received: invoke 274 bytes
rtmpdump: DEBUG: (object begin)
rtmpdump: DEBUG: Property: NULL
rtmpdump: DEBUG: (object begin)
rtmpdump: DEBUG: Property: <Name: level, STRING: status>
rtmpdump: DEBUG: Property: <Name: code, STRING: NetStream.Play.Stop>
rtmpdump: DEBUG: Property: <Name: description, STRING: Stopped playing public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4.>
rtmpdump: DEBUG: Property: <Name: details, STRING: public/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4>
rtmpdump: DEBUG: Property: <Name: clientid, STRING: qAAOwpJA>
rtmpdump: DEBUG: Property: <Name: reason, STRING: >
rtmpdump: DEBUG: (object end)
rtmpdump: DEBUG: (object end)
rtmpdump: DEBUG: HandleInvoke, server invoking <onStatus>
rtmpdump: DEBUG: HandleInvoke, onStatus: NetStream.Play.Stop
rtmpdump: DEBUG: Invoking deleteStream
rtmpdump: DEBUG: Got Play.Complete or Play.Stop from server. Assuming stream is complete
rtmpdump: DEBUG: RTMP_Read returned: 3655
rtmpdump: DEBUG: Closing connection.

Done. Saved 23271668 bytes to BBC_-_Co-op_supermarket_to_stop_selling_Nuts_magazine.flv

Tags: patch
Revision history for this message
Pigeon (x-launchpad-ubuntu) wrote :
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "Patch to fix bug downloading BBC videos" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
Revision history for this message
Pigeon (x-launchpad-ubuntu) wrote :

Er, yes, it is indeed a patch... :confused:

Revision history for this message
Brian Murray (brian-murray) wrote :

I was unable to recreate this using Saucy, which will become Ubuntu 13.10, which get-flash-videos version 1.25~git2012.06.27-1. The description mentions 11.04, is that really the release of Ubuntu you are using?

Revision history for this message
Pigeon (x-launchpad-ubuntu) wrote :

Yes, I rarely run the latest version of anything - I generally run the oldest version that will do what I want, to reduce clutter.

I have just tried again using get-flash-videos version 1.25~git2012.06.27-1 for Saucy freshly downloaded from the Ubuntu website, and the same URL. Without the patch, this does still fail in the exact same manner as in my original report.

With the patch applied, however, it succeeds - but today it retrieves an MP4 video over HTTP, whereas yesterday, for the same URL, it retrieved an FLV over RTMP, as shown above. This is the output from the patched version today:

$ get_flash_videos --debug http://www.bbc.co.uk/news/uk-23626232
2 plugins installed:
- Hulu.pm
- Itv.pm
Trying to open plugin ~/.get_flash_videos/plugins/Www.pm
Trying to open plugin ~/.get_flash_videos/plugins/Bbc.pm
Using method 'bbc' for http://www.bbc.co.uk/news/uk-23626232
Downloading http://www.bbc.co.uk/news/uk-23626232
-> GET http://www.bbc.co.uk/news/uk-23626232
<- 200 text/html text/html; charset=utf-8 (58275)
-> GET http://playlists.bbc.co.uk/news/uk-23626232A/playlist.sxml
<- 200 text/xml (4864)
-> GET http://open.live.bbc.co.uk/mediaselector/4/mtis/stream/p01dwkgh
<- 200 text/xml (5841)
Downloading http://news.downloads.bbc.co.uk.edgesuite.net/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4?at=oBWKGaMw3f12418bb84241a43680b069d772c307acacb8dd4e38ab4135340...
-> GET http://news.downloads.bbc.co.uk.edgesuite.net/mps_h264_hi/public/news/uk/1001000/1001225_h264_1500k.mp4?at=oBWKGaMw3f12418bb84241a43680b069d772c307acacb8dd4e38ab4135340
1001225_h264_1500k.mp4: 100% (22667.88 / 22667.88 KiB)<- 200 video/mp4 (0)

Done. Saved 23211909 bytes to 1001225_h264_1500k.mp4

Obviously something has changed on the BBC website as while the content of the video is the same, the format is different from when I posted the original report.

If you tried using the same URL as I did, then I would guess that maybe there was more than one change on the BBC website and at the time you tried to recreate it they were serving data which was compatible with the unpatched version. Or maybe you are not in the UK and it serves a different page outside the UK.

Revision history for this message
Brian Murray (brian-murray) wrote :

I am not in the UK.

Revision history for this message
Luite van Zelst (luitevanzelst) wrote :

Can confirm Pigeon's findings. However, there are several variations of BBC pages, seemingly based on
1) If you are seen to be connecting from UK ip address
2) Age of the BBC webpage.

giving variations with a href, param and .xml and .sxml.
I've tweaked pigeon's code by adding one more variation I've encountered

  elsif ($browser->content =~ /(http:[^":]+?playlist\.s?xml)/) {
    $playlist_xml = $1;
    $playlist_xml =~ s/\\//g;
  }

Think this is an upstream issue; have also reported my findings on
http://code.google.com/p/get-flash-videos/issues/detail?id=169

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.