mq_overview(7) contains outdated, inaccurate fs.mqueue.queues_max limit description

[Arto Bendiken]

The manpages 3.40-0.1ubuntu3 package in Ubuntu 12.10 contains an mq_overview(7) man page with outdated, severely inaccurate information regarding the /proc/sys/fs/mqueue/queues_max kernel limit, as follows:

"The default value for queues_max is 256; it can be changed to any value in the range 0 to INT_MAX."

This used to be true in Linux 3.2.x, as shipped in Ubuntu 12.04. However, it is no longer true with Linux 3.5.x, as shipped in 12.10.

For some inexplicable reason, upstream lowered the hard upper limit for POSIX message queues (fs.mqueue.queues_max) from INT_MAX to 1024 in the 3.5 kernel:

# sysctl -w fs.mqueue.queues_max=1025
sysctl: setting key "fs.mqueue.queues_max": Invalid argument
fs.mqueue.queues_max = 1025

You can verify the new, post-3.5 limit at:

http://lxr.free-electrons.com/source/ipc/mq_sysctl.c?v=3.5#L40
http://lxr.free-electrons.com/source/include/linux/ipc_namespace.h?v=3.5#L121

Please amend the man page so that the new limit is correctly documented. The undocumented change from INT_MAX queues to a mere 1024 queues is rather harsh, and it took quite some investigation to pin this down as an upstream kernel change.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Michael Kerrisk (mtk-manpages)]

Arto, clearly the man-page needs some fixing, and I'll look into it.

But, I am curious. Did this change cause some real-world applications to break?

[Michael Kerrisk (mtk-manpages)]

This appears to be the relavant change:

[Michael Kerrisk (mtk-manpages)]

Whoops. Hit post too soon on the last...

The change seems to be this ONE:

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=93e6f119c0ce8a1bba6e81dc8dd97d67be360844

[Michael Kerrisk (mtk-manpages)]

Also:
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=02967ea08ede0f8cc7e0526aedffdae65a099b07

[Arto Bendiken (artob)]

Michael, thanks for looking into this.

In answer to your question: yes, it caused the IPC mechanism in the database engine my company develops (dydra.com) to break, causing not a little aggravation. While we don't need millions of message queues, we had been relying on having at least a few thousand, and 1024 is just too low a hard limit. As we can't possibly require a custom-built kernel to run our software, we are now contemplating switching from POSIX to SysV message queues instead. Not a happy day...

[Michael Kerrisk (mtk-manpages)]

On Sun, Mar 17, 2013 at 12:57 PM, Arto Bendiken <email address hidden> wrote:
> Michael, thanks for looking into this.
>
> In answer to your question: yes, it caused the IPC mechanism in the
> database engine my company develops (dydra.com) to break, causing not a
> little aggravation. While we don't need millions of message queues, we
> had been relying on having at least a few thousand, and 1024 is just too
> low a hard limit. As we can't possibly require a custom-built kernel to
> run our software, we are now contemplating switching from POSIX to SysV
> message queues instead. Not a happy day...

Arto, I don't think you should be doing that. The upstream kernel
broke user space here, and I am sure that was not intended. I would
imagine that an upstream fix that repairs your problem would probably
be accepted. How about we pursue that line instead?

--
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Author of "The Linux Programming Interface"; http://man7.org/tlpi/

[Arto Bendiken (artob)]

Michael, time constraints mean that we'll likely have to put in place a SysV-based workaround in any case.

But sure, I'm willing to make the attempt to get this fixed upstream as well.

The hard limit remains the same from Linux 3.5 to 3.8, looks like:

http://lxr.free-electrons.com/source/include/linux/ipc_namespace.h?v=3.8#L121

I will defer to your authority regarding the best way to proceed ; a bug filing somewhere, or just an email to LKML?

PS. Just read the POSIX message queues sample chapter from your book. Looks like I'll have to get a hardcopy of TLPI to place on the shelf right next to my Stevens.

[Michael Kerrisk (mtk-manpages)]

Arto, it's the simple fix, but before I push it further, does the attached patch fix your problem?

At this point (ii.e., since LInux 3.8), I suspect that the limit that the kernel code currently tries to enforce on queues_max is anyway meaningless. Unprivileged users can nowadays use clone(CLONE_NEWUSER|CLONE_NEWIPC) to create an arbitrary number of new IPC namespaces, each of which can be populated with queues_max MQs. So the hard limit check doesn't really limit users at all (though, as you've discovered, it does inconvenience them).

[Arto Bendiken (artob)]

Michael, thanks for that; the patch looks good, but I will make some time today to build a patched kernel on a virtual machine and get back to you with a confirmation.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "mqueue_limit.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Arto Bendiken (artob)]

Michael, I tested your patch against Ubuntu 12.10's 3.5.0-26-generic kernel, applied after `apt-get source`. Your patch does apply and build cleanly (offset 1 line compared to the 3.9 sources).

However, it unfortunately does not fix the issue. sysctl still cannot be used to raise the message queue limit above 1024. Given this, ipc_ns->mq_queues_max is still initialized to DFLT_QUEUESMAX (256) and ultimately limited by HARD_QUEUESMAX (1024), even if that's not being tested for directly in mqueue_create().

I've attached a simpler, one-line patch that I've confirmed does fully solve the problem for us; it just raises the HARD_QUEUESMAX constant from 1K to 1M. I believe this would be the least intrusive change to make, just restoring a sufficient amount of the capacity that the kernel used to provide back in the pre-3.5 days of an INT_MAX limit.

[Arto Bendiken (artob)]

Michael, what do you think should be the next step here? Is raising HARD_QUEUESMAX back upwards feasible, or should the patch be more in line with what you began formulating?

[Michael Kerrisk (mtk-manpages)]

> However, it unfortunately does not fix the issue.
> sysctl still cannot be used to raise the message
> queue limit above 1024. Given this, ipc_ns->mq_queues_max
> is still initialized to DFLT_QUEUESMAX (256) and
> ultimately limited by HARD_QUEUESMAX (1024),
> even if that's not being tested for directly in
> mqueue_create().

Arto,

Sorry for the delayed follow up., The point of the patch was *not* to allow the sysctl to be changed. Rather, it was to allow a *privileged* process to override the limit. Is you DB engine running as a privileged process? If not, then I suppose a somewhat different patch is in order.

[Arto Bendiken (artob)]

Michael,

We do run the database software as unprivileged users, yes. Previously, our installation instructions have advised configuring appropriate values in /etc/sysctl.d/*.conf and /etc/security/limits.d/*.conf for the system limits and per-user limits, respectively. After kernel 3.5+, that strategy obviously doesn't work so well any longer.

It would be really good to bump the kernel hard limit upward some from the much-too-harsh 1024, even if the patch also otherwise improves handling for privileged users.

[Arto Bendiken (artob)]

Just to follow up on this, we have since abandoned the use of POSIX message queues altogether so this issue is no longer an ongoing concern for us.

Still, the mq_overview(7) man page continues to refer to the outdated, inaccurate queues_max hard limit (INT_MAX instead of 1024) at least as of Ubuntu 13.10.

[Michael Kerrisk (mtk-manpages)]

On 02/19/2014 11:01 AM, Arto Bendiken wrote:
> Just to follow up on this, we have since abandoned the use of POSIX
> message queues altogether so this issue is no longer an ongoing concern
> for us.
>
> Still, the mq_overview(7) man page continues to refer to the outdated,
> inaccurate queues_max hard limit (INT_MAX instead of 1024) at least as
> of Ubuntu 13.10.

Yes, it's been on my long TODO list to follow this up. Doc will get fixed,
and I hope the kernel too. Tell me, was this limitation the reason to
abandon PMQs?

--
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

[Arto Bendiken (artob)]

Hi Michael,

Yes, we couldn't operate within the constraints of a queues_max=1024 kernel hard limit, and could not possibly require our customers to install a custom-patched kernel, so we had to abandon POSIX message queues altogether and switch to other forms of IPC. (Your book was helpful in this regard, by the way, in gaining a good overview and understanding of all available forms of IPC.)

Thanks for following up,
Arto

[Davidlohr Bueso (davidlohr)]

This issue has been addressed upstream and will be available in Linux 3.14 -- should be backported to stable kernels as well.

Commit f3713fd9 [https://lkml.org/lkml/2014/2/9/224]

[Sebastien Bacher (seb128)]

The manpage is correct in xenial, closing the bug