iax2 out of call numbers no more space

[iBro X]

System:
Linux asterisk 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Description: Ubuntu 16.04.3 LTS
Release: 16.04

Problematic package:
asterisk:
  Installed: 1:13.1.0~dfsg-1.1ubuntu4.1
  Candidate: 1:13.1.0~dfsg-1.1ubuntu4.1
  Version table:
 *** 1:13.1.0~dfsg-1.1ubuntu4.1 500
        500 http://id.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
        100 /var/lib/dpkg/status
     1:13.1.0~dfsg-1.1ubuntu4 500
        500 http://id.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages

Description:
Our Asterisk installation has been crashing several times with IAX2 throwing out the following error messages:

[Jan 1 17:57:50] WARNING[21043] chan_iax2.c: Out of call numbers
[Jan 1 17:57:50] WARNING[21043] chan_iax2.c: No more space
[Jan 1 17:57:58] WARNING[21039] chan_iax2.c: Out of call numbers
[Jan 1 17:57:58] WARNING[21039] chan_iax2.c: No more space
[Jan 1 17:58:00] WARNING[21044] chan_iax2.c: Out of call numbers
[Jan 1 17:58:00] WARNING[21044] chan_iax2.c: No more space
[Jan 1 17:58:05] WARNING[21041] chan_iax2.c: Out of call numbers
[Jan 1 17:58:05] WARNING[21041] chan_iax2.c: No more space
[Jan 1 17:58:14] WARNING[21042] chan_iax2.c: Out of call numbers
[Jan 1 17:58:14] WARNING[21042] chan_iax2.c: No more space
[Jan 1 17:58:16] WARNING[21045] chan_iax2.c: Out of call numbers
[Jan 1 17:58:16] WARNING[21045] chan_iax2.c: No more space
[Jan 1 17:58:22] WARNING[21038] chan_iax2.c: Out of call numbers
[Jan 1 17:58:22] WARNING[21038] chan_iax2.c: No more space
[Jan 1 17:58:25] WARNING[21043] chan_iax2.c: Out of call numbers
[Jan 1 17:58:25] WARNING[21043] chan_iax2.c: No more space
[Jan 1 17:58:39] WARNING[21044] chan_iax2.c: Out of call numbers
[Jan 1 17:58:39] WARNING[21044] chan_iax2.c: No more space
[Jan 1 17:58:40] WARNING[21041] chan_iax2.c: Out of call numbers
[Jan 1 17:58:40] WARNING[21041] chan_iax2.c: No more space
[Jan 1 17:58:48] WARNING[21039] chan_iax2.c: Out of call numbers
[Jan 1 17:58:48] WARNING[21039] chan_iax2.c: No more space
[Jan 1 17:58:50] WARNING[21038] chan_iax2.c: Out of call numbers
[Jan 1 17:58:50] WARNING[21038] chan_iax2.c: No more space
[Jan 1 17:58:55] WARNING[21037] chan_iax2.c: Out of call numbers
[Jan 1 17:58:55] WARNING[21037] chan_iax2.c: No more space

The last time it happened, I issue 'iax2 show callnumber usage' and it shows that I am out of Regular callno:

Address Callno Usage Callno Limit
172.105.250.254:4569 4 2048
172.16.171.18:4569 1 2048
172.16.182.1:4569 4 2048
172.16.10.11:4569 1645 2048
121.202.13.78:4569 1644 2048
172.16.10.17:4569 1643 2048
172.16.128.1:4569 1 2048
172.105.252.254:4569 4 2048
172.16.10.20:4569 1563 2048
119.18.155.242:4569 3 2048
172.16.171.94:4569 4 2048
172.16.185.1:4569 1 2048
172.16.10.13:4569 1642 2048
172.16.10.16:4569 1642 2048
172.16.10.25:4569 1642 2048
172.16.10.12:4569 1642 2048
172.16.10.15:4569 1648 2048
172.16.10.18:4569 7 2048
172.16.10.21:4569 1642 2048

Non-CallToken Validation Callno Limit: 8192
Non-CallToken Validated Callno Used: 0
Total Available Callno: 16384
Regular Callno Available: 0
Trunk Callno Available: 16384

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: asterisk 1:13.1.0~dfsg-1.1ubuntu4.1
ProcVersionSignature: Ubuntu 4.4.0-104.127-generic 4.4.98
Uname: Linux 4.4.0-104-generic x86_64
NonfreeKernelModules: dahdi_vpmadt032_loader
ApportVersion: 2.20.1-0ubuntu2.13
Architecture: amd64
Date: Thu Jan 4 10:38:11 2018
InstallationDate: Installed on 2017-10-26 (69 days ago)
InstallationMedia: Ubuntu-Server 16.04.3 LTS "Xenial Xerus" - Release amd64 (20170801)
JournalErrors:
 Error: command ['journalctl', '-b', '--priority=warning', '--lines=1000'] failed with exit code 1: Hint: You are currently not seeing messages from other users and the system.
       Users in the 'systemd-journal' group can see all messages. Pass -q to
       turn off this notice.
 No journal files were opened due to insufficient permissions.
SourcePackage: asterisk
UpgradeStatus: No upgrade log present (probably fresh install)

[iBro X (ibro-x)]

I do not know if this is relevant, but there a kernel message which seems suspicious to me:

[499321.365191] traps: asterisk[17039] trap invalid opcode ip:7fb46af2dedc sp:7fb3e4c0b890 error:0 in libpthread-2.23.so[7fb46af21000+18000]

[Christian Ehrhardt  (paelzer)]

Function get_unused_callno issues "Out of call numbers"
And Function __find_callno issues "No more space" if the former fails
So this is no usual out of (disk) space or such.

[Christian Ehrhardt  (paelzer)]

I'm really no asterisk expert but there are two cases:
a) you are really out of numbers (not a bug, just as it is)
b) asterisk thinks you are out of numbers due to a bug e.g. not freeing unused correctly

Could you confirm that we are looking at (b) in this case?

Also how easily is this reproducible for you?
If you can kind of intentionally trigger it I'd recommend you might try one of the newer versions available in newer Ubunutu releases - if those work reliably that might confirm that there already is a fix in the asterisk project that we have to look for to backport.

For the illegal opcode - there are (to my surprise) several of those issues found by search engines. Those issues usually are super rare - only if either the code is totally wrong, compiled against wrong libs/headers or virtual machines which do not provide all expected cpu features.

It sounds similar to [1], but that code is in the version you run.
Also [2] is similar but very old as well.
Maybe you find some tests/workarounds there still?

TL;DR - as unfortunate as it is for now I can't differ this between a bug or a config issue.
Waiting for your feedback - hopefully this is somewhat reproducible to be debuggeble by you.

[1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=611628
[2]: http://lists.digium.com/pipermail/asterisk-users/2008-September/218759.html

[Christian Ehrhardt  (paelzer)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Since it seems likely to me that this is a local configuration problem, rather than a bug in Ubuntu, I'm marking this bug as Incomplete.

If indeed this is a local configuration problem, you can find pointers to get help for this sort of problem here: http://www.ubuntu.com/support/community

Or if you believe that this is really a bug, then you may find it helpful to read "How to report bugs effectively" http://www.chiark.greenend.org.uk/~sgtatham/bugs.html. We'd be grateful if you would then provide a more complete description of the problem, explain why you believe this is a bug in Ubuntu rather than a problem specific to your system, and then change the bug status back to New.