xargs complains command line is too long, but kernel allows command lines of arbitrary length

[Reuben Thomas]

Binary package hint: findutils

I am getting

xargs: command line too long

errors, but I am generating command lines of only a few hundred kilobytes in length and the kernel is perfectly capable of handling these, as I find if I construct the command-line manually.

ProblemType: Bug
Architecture: i386
Date: Fri Jul 18 19:50:49 2008
Dependencies:
 libgcc1 1:4.2.3-2ubuntu7
 gcc-4.2-base 4.2.3-2ubuntu7
 libc6 2.7-10ubuntu3
DistroRelease: Ubuntu 8.04
NonfreeKernelModules: ath_hal
Package: findutils 4.2.32-1ubuntu2
PackageArchitecture: i386
ProcEnviron:
 SHELL=/bin/bash
 PATH=/usr/local/sbin:/sbin:/usr/sbin:/usr/local/epocemx/bin:/home/username/bin:/home/username/local/i686/bin:/home/username/local/bin:/home/username/.luarocks/bin:/home/username/Work/Adsensus/svn/nancy/trunk:/home/username/Work/Adsensus/svn/adsensus/trunk:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
 LANG=en_GB.UTF-8
SourcePackage: findutils
Uname: Linux 2.6.24-19-generic i686

[Jean-Baptiste Lallement (jibel)]

Thanks for your report.

Please, can you post a testcase and tell us in details how to reproduce this issue.
This will greatly help us to track down your problem. Thanks.

[Reuben Thomas (rrt)]

xargs -0 touch < foo

Where foo is a file containing no NUL bytes that is longer than 128Kb. If the file is shorter, I get an error from touch instead, which seems reasonable (file name too long). But why does xargs complain?

In fact, I got the error message slightly wrong, it is:

"argument line too long"

[Jean-Baptiste Lallement (jibel)]

confirming.

== testcase ==
for f in `seq 1 20000`;echo -n "xxxxxxxx "; done | xargs -0

Only happens with -0 which means the input line is processed as one long line.

[Jean-Baptiste Lallement (jibel)]

Are you using a stock kernel or a customized one.

Please can you provide the output of uname -a

Thanks.

[Reuben Thomas (rrt)]

Stock kernel:

Linux canta 2.6.24-19-generic #1 SMP Wed Jun 18 14:43:41 UTC 2008 i686 GNU/Linux

[Jean-Baptiste Lallement (jibel)]

In Ubuntu the maximum page allocation for env+arg is hard coded and limited to 32 pages of 4KB.

If you look at the include/linux/binfmts.h file, you will find the following near the top:

/*
 * MAX_ARG_PAGES defines the number of pages allocated for arguments
 * and envelope for the new program. 32 should suffice, this gives
 * a maximum env+arg of 128kB w/4KB pages!
 */
#define MAX_ARG_PAGES 32

[Reuben Thomas (rrt)]

This is not what I find. Instead, I have:

/*
 * These are the maximum length and maximum number of strings passed to the
 * execve() system call. MAX_ARG_STRLEN is essentially random but serves to
 * prevent the kernel from being unduly impacted by misaddressed pointers.
 * MAX_ARG_STRINGS is chosen to fit in a signed 32-bit integer.
 */
#define MAX_ARG_STRLEN (PAGE_SIZE * 32)
#define MAX_ARG_STRINGS 0x7FFFFFFF

so, it seems that there is a maximum string length, but not a maximum overall argument length. Still, you can close this bug.

[Jean-Baptiste Lallement (jibel)]

It's defined a few lines below at line 32 on my version.

black:$ grep -n MAX_ARG_PAGES /usr/src/linux-headers-2.6.26-4/include/linux/binfmts.h
32:# define MAX_ARG_PAGES 32

You can safely increase this value and recompile the kernel to give more space to arguments.

Closing this report as requested.

Thanks for your time.

[Reuben Thomas (rrt)]

You're quoting from a non-stock (or at least, non-hardy) kernel. Is this limit re-established in later (Ubuntu?) kernels? If so, I'm deeply disappointed, as this was a great innovation in Linux. If it's Ubuntu-specific, I'd like to know, so I can file a bug against the relevant kernel package.

[Jean-Baptiste Lallement (jibel)]

Well. I've been wrong on this one. I'm afraid that the kernel has nothing to do with this.

The limit for find and xargs is hardcoded in lib/buildcmd.c in bc_use_sensible_arg_max and can be at build time.

Here is the relevant part of the changelog and more details can be found in the README.

2007-07-31 Eric Blake <email address hidden>

 Allow choice of default arg size, Savannah bug #20594.
 * configure.ac (DEFAULT_ARG_SIZE): Check environment for a default
 size override.
 * lib/buildcmd.c (bc_use_sensible_arg_max): Use default size from
 configure, if requested.
 * README (DEFAULT_ARG_SIZE): Mention the ability to tune this at
 configure time.
 * NEWS: Document the change.

I'm marking this report "confirmed" .