Image abnmormally stretched when streaming with transcode activated (Integer overflow in aspect ratio calculation)

[Nicola Ferralis]

Binary package hint: vlc

Using VLC 1.1.4-1 on Ubuntu 10.10. When I try to stream and transcode from capturing with a webcam, the image in the stream is always stretched vertically. Note that if I select to stream without transcoding, the image, in either case, looks normal. Please keep in mind that this bug seems to be only reproducible with Ubuntu. In Windows everything works as expected. I tried several webcams, with similar results. As it is, streaming from a capture device like a webcam is completely unusable.

The cause of the bug (a comprehensive description is given in the upstream report) is due to a integer overflow during transcoding. Basically, the aspect ratio for a 640x320 webcam is given in absolute terms, for example 276480000:276480000 which is equivalent to 1:1, but when multiplied by 640 the result is > 32 bits.

The attached patch essentially prevents the overflow by reducing the value of the aspect ratio within 32 bit and by assigning a working value for the aspect ratio of the image after decoding.

TEST CASE to reproduce:
0. Make sure you have a webcam attached.
1. Launch vlc
2. press CTRL+C for capture from the webcam using v4l2
3. press ALT+S for streaming
4. select stream to a file (but http will do also)
5. Important: make sure you select a transcoding option. Anyone is
good, other than OGG (where this bug seem not to apply).
6. Start the stream and wait a few minutes, before turning the stream OFF.

Expected: the file (or http stream) should have an image with the
correct aspect ratio.

Actual: the video is deformed (in my case vertically elongated.

The patch has been extensively tested on test builds on VLC 1.1.4-1ubuntu1 in maverick. No collateral effects are expected.

[Nicola Ferralis (feranick)]

Images depicting the abnormal behaviour attached to the upstream bug report:

https://trac.videolan.org/vlc/ticket/4312

[Nicola Ferralis (feranick)]

The file actually plays just fine. However it will not play (in any other player includinc VLC) UNTIL after the session with VLC has been closed. It would be great if the all process would be more transparent...

[Nicola Ferralis (feranick)]

Marked invalid upstream. Marking invalid here too.

[Nicola Ferralis (feranick)]

Attached working patch from upstream. Tested on vlc 1.1.5ubuntu1 in natty. Test builds available here:

https://launchpad.net/~natty-bleed/+archive/ppa

[Nicola Ferralis (feranick)]

Fix released (with the attached patch) upstream, 1.1.5 branch and 1.2 trunk.

[Benjamin Drung (bdrung)]

uploaded to natty with these changes:
* version changed to 1.1.5-1ubuntu2
* "Closes" changed to "LP"
* added DEP-3 header to patch

[Benjamin Drung (bdrung)]

Please prepare a SRU [1] if you want the fix in lucid, too.

[1] https://wiki.ubuntu.com/StableReleaseUpdates

[Launchpad Janitor (janitor)]

This bug was fixed in the package vlc - 1.1.5-1ubuntu2

---------------
vlc (1.1.5-1ubuntu2) natty; urgency=low

  * Fixed wrong aspect ratio in transcoded image from webcam. (LP: #672304)
 -- Nicola Ferralis <email address hidden> Fri, 19 Nov 2010 00:15:09 +0100

[Nicola Ferralis (feranick)]

Version 1.0.6-1ubuntu1.2 in Lucid is NOT affected. However version 1.1.4-1ubuntu1 in Maverick is affected.

I am preparing a SRU for Maverick.

[Benjamin Drung (bdrung)]

Oh yes, I meant maverick, not lucid. :)

[Nicola Ferralis (feranick)]

Uploaded patch and debdiff for maverick-proposed. Changelog below:

vlc (1.1.4-1ubuntu2) maverick-proposed; urgency=low

  * Fixed wrong aspect ratio in transcoded image from webcam. (LP: #672304)

 -- Nicola Ferralis <email address hidden> Thu, 18 Nov 2010 21:47:28 -0800

[Nicola Ferralis (feranick)]

Apologizes, I don't see the "nominate for release" link... Can someone add the nomination for me, please?

[Micah Gersten (micahg)]

Thank you for preparing this patch. Could you please change the version to 1.1.4-1ubuntu1.1 and also add the appropriate DEP-3 headers to the patch?

[Micah Gersten (micahg)]

Please subscribe sponsors again when ready.

[Nicola Ferralis (feranick)]

Done. New changelog:

vlc (1.1.4-1ubuntu1.1) maverick-proposed; urgency=low

  * Fixed wrong aspect ratio in transcoded image from webcam. (LP: #672304)

 -- Nicola Ferralis <email address hidden> Fri, 19 Nov 2010 01:12:25 -0800

[Nicola Ferralis (feranick)]

(added DEP-3 headers)

[Benjamin Drung (bdrung)]

You can use "Bug-Ubuntu: https://launchpad.net/bugs/672304" for having shorter links.

[Nicola Ferralis (feranick)]

Thanks. Re-uploaded with revised DEP-3.

(I apologize for the multiple uploads. I am new at this.)

[Micah Gersten (micahg)]

Nicola, you're doing great. Thank you for the update. I will upload after testing the building locally.

[Benjamin Drung (bdrung)]

Micah, you should say that you have uploaded it to proposed.

[Micah Gersten (micahg)]

I have uploaded it to -proposed :)

[Martin Pitt (pitti)]

Accepted vlc into maverick-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Nicola Ferralis (feranick)]

Version 1.1 in proposed tested and verified to work just fine.

tag should be changed to --verification-done

[Nicola Ferralis (feranick)]

Sorry, I meant version 1.1.4ubuntu1.1.

[Launchpad Janitor (janitor)]

This bug was fixed in the package vlc - 1.1.4-1ubuntu1.1

---------------
vlc (1.1.4-1ubuntu1.1) maverick-proposed; urgency=low

  * Fixed wrong aspect ratio in transcoded image from webcam. (LP: #672304)
 -- Nicola Ferralis <email address hidden> Fri, 19 Nov 2010 01:12:25 -0800