SIGSEGV in x265_10bit::RateControl::rateEstimateQscale()

[Keith Helms]

Description: Ubuntu 20.04.1 LTS
Release: 20.04

Package libx265-179 version 3.2.1-1build1

Happens randomly under both FFMpeg and HandBrakeCLI. An encode of the same file will crash one time, but work the next time.

Excerpt from the crash file:

SegvAnalysis:
 Segfault happened at: 0x7f0613a1f3a0 <_ZN10x265_10bit11RateControl18rateEstimateQscaleEPNS_5FrameEPNS_16RateControlEntryE+112>: mov (%r8),%rcx
 PC (0x7f0613a1f3a0) ok
 source "(%r8)" (0x00000000) not located in a known VMA region (needed readable region)!
 destination "%rcx" ok
 Stack memory exhausted (SP below stack segment)
SegvReason: reading NULL VMA
SourcePackage: ffmpeg
Stacktrace:
 #0 0x00007f0613a1f3a0 in x265_10bit::RateControl::rateEstimateQscale(x265_10bit::Frame*, x265_10bit::RateControlEntry*) () at /lib/x86_64-linux-gnu/libx265.so.179
 #1 0x00007f0613a21678 in x265_10bit::RateControl::rateControlStart(x265_10bit::Frame*, x265_10bit::RateControlEntry*, x265_10bit::Encoder*) () at /lib/x86_64-linux-gnu/libx265.so.179
 #2 0x00007f0613a12fc1 in x265_10bit::FrameEncoder::compressFrame() () at /lib/x86_64-linux-gnu/libx265.so.179
 #3 0x00007f0613a1592a in x265_10bit::FrameEncoder::threadMain() () at /lib/x86_64-linux-gnu/libx265.so.179
 #4 0x00007f06139ec52e in () at /lib/x86_64-linux-gnu/libx265.so.179
 #5 0x00007f061a321609 in start_thread (arg=<optimized out>) at pthread_create.c:477
         ret = <optimized out>
         pd = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139663137658624, 8136522380067276986, 140736498594622, 140736498594623, 140736498594624, 139663137653696, -8151220019210155846, -8149668580039915334}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = 0
 #6 0x00007f061a248293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
StacktraceAddressSignature: /usr/bin/ffmpeg:11:/usr/lib/x86_64-linux-gnu/libx265.so.179+4f63a0:/usr/lib/x86_64-linux-gnu/libx265.so.179+4f8678:/usr/lib/x86_64-linux-gnu/libx265.so.179+4e9fc1:/usr/lib/x86_64-linux-gnu/libx265.so.179+4ec92a:/usr/lib/x86_64-linux-gnu/libx265.so.179+4c352e:/usr/lib/x86_64-linux-gnu/libpthread-2.31.so+2609:/usr/lib/x86_64-linux-gnu/libc-2.31.so+fd293
StacktraceTop:
 x265_10bit::RateControl::rateEstimateQscale(x265_10bit::Frame*, x265_10bit::RateControlEntry*) () at /lib/x86_64-linux-gnu/libx265.so.179
 x265_10bit::RateControl::rateControlStart(x265_10bit::Frame*, x265_10bit::RateControlEntry*, x265_10bit::Encoder*) () at /lib/x86_64-linux-gnu/libx265.so.179
 x265_10bit::FrameEncoder::compressFrame() () at /lib/x86_64-linux-gnu/libx265.so.179
 x265_10bit::FrameEncoder::threadMain() () at /lib/x86_64-linux-gnu/libx265.so.179
 () at /lib/x86_64-linux-gnu/libx265.so.179

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: x265 3.2.1-1build1
ProcVersionSignature: Ubuntu 5.4.0-58.64-generic 5.4.73
Uname: Linux 5.4.0-58-generic x86_64
NonfreeKernelModules: wl
ApportVersion: 2.20.11-0ubuntu27.14
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: XFCE
Date: Sun Jan 3 18:49:17 2021
InstallationDate: Installed on 2020-12-26 (8 days ago)
InstallationMedia: Xubuntu 20.04.1 LTS "Focal Fossa" - Release amd64 (20200731)
ProcEnviron:
 LANGUAGE=en_US
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: x265
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.