Comment 10 for bug 4229

The BadAlloc comes with the video just doesn't have the resources to give the program; either there isn't enough free RAM on the video card for an Xvideo buffer of the size requested, or the Xv port cannot decode images that large.

It really should be up to totem to realise this, back-off and try another combination (eg. software rendering).