getstream quits with SIGABRT when ANY of the tuners loses signal

[MMlosh]

I am running getstream on 4 usb tuners (one instance for all)

When any of the tuners loses signal, getstream aborts with SIGABRT.
That, obviously, interrupts all streams, even those streamed from good tuners.

I found an important information in the changelog:
"
I added a guardian thread which basically monitors if the libevent loop in the main program is still triggering events. If not we'll send a SIGABRT and when you are lucky and have set a core ulimit we'll have something to debug afterwards.
"

This has been happening across quite a few gestream and ubuntu versions
The current version originates from Quantal, (20081204-1ubuntu4)

Note: I am running it on ARM, not x86

[MMlosh (mmlosh)]

Just as I expected. Adding new tuner to capture a weaker dvb-t multiplex causes a sigabrt every now and then, usually at least once per hour, causing disruptions to all multiplexes.

[MMlosh (mmlosh)]

According to the source code the SIGABRT watchdog thread should only be active when compiled "with DEBUG"
That also produces a notice on every http stream start and end (also matches my observations)

I'll try building it without debug to see whether it's really locking up, or only the watchdog misbehaving.

[MMlosh (mmlosh)]

I disabled the watchdog.
Temporary signal outage on one tuner does not affect other tuners for at least 30s.

If the multiplex from the tuner was streamed somewhere, a warning appears on the console, saying "demux: read in dvr_read returned with errno 75"

I wonder whether the libevent loop gets actually stuck, or just delayed.

[MMlosh (mmlosh)]

correction:
If the multiplex from the *affected (signal loss)* tuner was streamed somewhere.....

Well, the watchdog is really necessary.
The event loop gets stuck eventually when one of the tuners has poor signal (and perhaps when I attempt to watch it?).
So far it seems that increasing the watchdog timeout from 1s to 2s reduces the abort rate dramatically.
That is, of course, no actual solution and could prevent debugging attempts.

[MMlosh (mmlosh)]

UPDATE: Progress!

The stall was traced to a the "while" loop in dvr_input_ts ( dvr.c:68 )
When psi_reassemble does not increase "off", it gets called again with the same arguments -> inf-loop

I believe that that can happen only when section->len == 0.
I have no idea where is that coming from nor why.
  Getstream itself? Kernel? USB DVBT tuner's firmware? Tuner's hardware? Actual content of the signal?

The .patch attached does _NOT_ fix the cause of the aborts.
It only makes sure that psi_reassemble either makes progress or returns errors and prevents the inf-loop.
I am not even sure that "PSI_RC_LENFAIL" is the correct return code for the situation. PSI_RC_NOPAYLOAD? PSI_RC_CORRUPT?

I also attach a pack with on-abor/lockup stacktraces I collected.

I'll let it run for a week and see, whether it actually worked.
So far the attached not-fix successfully replaced all "abort" lines with "warning" lines in the logs, making getstream useful again.

I don't think that the patch is ready for upstream, but at least it allows me to use getstream
I hope it helps someone more involved than me to actually fix the issue.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Not a real fix, but at least it removes the crash symptom." seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[MMlosh (mmlosh)]

There was a complete signal outage yesterday + a power outage recently. I don't know which messages should be atributed to which cause.
Getstream is definitely not immune to complete signal outage, not even with my hackpatch.
I *guess* that it would be fine with the watchdog disabled.. (just wait for the signal to come back)