dd: cryptic error message when bs=<unreasonably large value>

[Imre Péntek]

Binary package hint: coreutils

imi@most:~$ true|dd bs=1 count=3G
0+0 records in
0+0 records out
0 bytes (0 B) copied, 4.0228e-05 s, 0.0 kB/s
imi@most:~$ true|dd bs=3G count=1
dd: invalid number `3G'

however, according to man DD:
       BLOCKS and BYTES may be followed by the following multiplicative suf‐
       fixes: c =1, w =2, b =512, kB =1000, K =1024, MB =1000*1000, M
       =1024*1024, xM =M GB =1000*1000*1000, G =1024*1024*1024, and so on for
       T, P, E, Z, Y.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: coreutils 7.4-2ubuntu2
ProcVersionSignature: Ubuntu 2.6.32-22.36-generic 2.6.32.11+drm33.2
Uname: Linux 2.6.32-22-generic i686
Architecture: i386
Date: Thu Jun 10 01:37:41 2010
ProcEnviron:
 PATH=(custom, user)
 LANG=hu_HU.utf8
 SHELL=/bin/bash
SourcePackage: coreutils

[Dave Gilbert (ubuntu-treblig)]

You're on 32bit, on 64bit this seems to work. My guess is that there is a limit to the block size available on 32bit (if that limit was 2G or just under it wouldn't surprise me), on 64bit there is a similar limit somewhere :

dg@major:~$ true | dd bs=3G count=1
0+0 records in
0+0 records out
0 bytes (0 B) copied, 2.1008e-05 s, 0.0 kB/s
dg@major:~$ true | dd bs=3E count=1
dd: memory exhausted
dg@major:~$ true | dd bs=3Z count=1
dd: invalid number `3Z'

I can see an argument that there might want to put a better error message in, or state that the bs is dependent on memory
size.

Dave

[Imre Péntek (pentek-imre)]

Maybe it is becouse I am on a 32bit system, but anyways what I am speaking about is dd behaviour and what is documented in man dd is inconsistent to each other. the resolution would be to change any of the two to match. (however I'd better like dd to handle bs=3G)

[C de-Avillez (hggdh2)]

(Jim Meyering wrote, on the coreutils-bugs mailing list:

Dave Gilbert wrote:
> You're on 32bit, on 64bit this seems to work. My guess is that there is
> a limit to the block size available on 32bit (if that limit was 2G or
> just under it wouldn't surprise me), on 64bit there is a similar limit
> somewhere :
>
> dg@major:~$ true | dd bs=3G count=1
> 0+0 records in
> 0+0 records out
> 0 bytes (0 B) copied, 2.1008e-05 s, 0.0 kB/s
> dg@major:~$ true | dd bs=3E count=1
> dd: memory exhausted
> dg@major:~$ true | dd bs=3Z count=1
> dd: invalid number `3Z'
>
> I can see an argument that there might want to put a better error message in, or state that the bs is dependent on memory
> size.

Thanks for the report.

dd is trying to tell you not to specify such a large buffer size.
A buffer size larger than a few megabytes is not normally useful.

That diagnostic is not helpful indeed, but it was easy to fix,
so I'll probably apply something like the following upstream.
With the new diagnostic, you can see that dd tries (and fails)
to allocate output buffers of the specified size:

    $ :|./dd bs=3EiB
    ./dd: failed to allocate an input buffer of size 3.0 EiB
    [Exit 1]
    $ :|./dd obs=3EiB
    ./dd: failed to allocate an output buffer of size 3.0 EiB
    [Exit 1]

Making dd give a better diagnostic for e.g., "bs=3G" on an i686
system might be more work than it's worth.

[C de-Avillez (hggdh2)]

For the record, this is the patch Jim wrote (which was inlines in his email to debbugs.gnu.org). I believe there was a conflict of emails -- Jim probably used a different email than the one he has registered on LP.

I am tagging this 'patch-upstreaminput' since Jim is expecting feedback from the list before committing. This fix will *probably be available on next upstream stable (8.6?).

Thank you, Jim.

[Imre Péntek (pentek-imre)]

vasárnap 13 június 2010 23:23:39 dátummal Ön az alábbiakat írta:
> Making dd give a better diagnostic for e.g., "bs=3G" on an i686
> system might be more work than it's worth.
Well, this way it's okay that dd does not accept 3G as block size, but then
man dd would document that according to the appropriate behavior.
Like this:
BYTES may be followed by the following multiplicative suffixes: c =1, w =2,
b=512, kB =1000, K =1024, MB =1000*1000, M=1024*1024, xM =M
GB=1000*1000*1000, G =1024*1024*1024, and so on for T, P, E, Z, Y.
The same applies to BLOCKS with the practical exception your
machine(?)/kernel(?)/... needs to be able to allocate memory for (at
least(???)) one(?) block, so, you can expect not to be able to use block sizes
greater or equal to 2G on a 32 bit architecture, and 16E on a 64 bit
architecture.

or something like this. what I read now in man dd is a partial documentation
which would (and had) me expect bs=3G to be OK in all cases, however it isn't
ok in my case.

Thank you for your reply anyways.
--
Üdvözlettel: Ifj. Péntek Imre
E-Mail: <email address hidden>

[Imre Péntek (pentek-imre)]

This is what I sent to C de-Avillez some minutes ago (somehow it seemed like he sent me a private fwd email). so copypaste:

> Making dd give a better diagnostic for e.g., "bs=3G" on an i686
> system might be more work than it's worth.
Well, this way it's okay that dd does not accept 3G as block size, but then
man dd would document that according to the appropriate behavior.
Like this:
BYTES may be followed by the following multiplicative suffixes: c =1, w =2,
b=512, kB =1000, K =1024, MB =1000*1000, M=1024*1024, xM =M
GB=1000*1000*1000, G =1024*1024*1024, and so on for T, P, E, Z, Y.
The same applies to BLOCKS with the practical exception your
machine(?)/kernel(?)/... needs to be able to allocate memory for (at
least(???)) one(?) block, so, you can expect not to be able to use block sizes
greater or equal to 2G on a 32 bit architecture, and 16E on a 64 bit
architecture.

or something like this. what I read now in man dd is a partial documentation
which would (and had) me expect bs=3G to be OK in all cases, however it isn't
ok in my case.

Thank you for your reply anyways.

[Jim Meyering (meyering)]

Imre Péntek wrote:
> vasárnap 13 június 2010 23:23:39 dátummal Ön az alábbiakat írta:
>> Making dd give a better diagnostic for e.g., "bs=3G" on an i686
>> system might be more work than it's worth.
> Well, this way it's okay that dd does not accept 3G as block size, but then
> man dd would document that according to the appropriate behavior.
> Like this:
> BYTES may be followed by the following multiplicative suffixes: c =1, w =2,
> b=512, kB =1000, K =1024, MB =1000*1000, M=1024*1024, xM =M
> GB=1000*1000*1000, G =1024*1024*1024, and so on for T, P, E, Z, Y.
> The same applies to BLOCKS with the practical exception your
> machine(?)/kernel(?)/... needs to be able to allocate memory for (at
> least(???)) one(?) block, so, you can expect not to be able to use block sizes
> greater or equal to 2G on a 32 bit architecture, and 16E on a 64 bit
> architecture.
>
> or something like this. what I read now in man dd is a partial documentation
> which would (and had) me expect bs=3G to be OK in all cases, however it isn't
> ok in my case.

Thanks for the suggestion, but we can't use that, since
dd --help content has to stay small and generally relevant.
[btw, the man page is generated from dd --help's output. ]
Using such large buffer sizes is neither common nor worth
documenting in a place like --help.

To improve the "real" documentation -- that's always welcome --
you would add to "info dd" (patch doc/coreutils.texi in the sources)
saying something like "Any block size you specify via bs=, ibs=, obs=, cbs=
should not be too large --- values larger than a few megabytes
are generally wasteful or (as in the gigabyte..exabyte case) downright
counterproductive or error-inducing."

While I deliberately do not mention the actual limits above,
note that they're about half of the numbers you listed: ~1GiB and ~8EiB

I went ahead and wrote the patch:

From d8e0d2d0efba4fa9c5bc1c29bf7e0028dd292237 Mon Sep 17 00:00:00 2001
From: Jim Meyering <email address hidden>
Date: Mon, 14 Jun 2010 10:15:57 +0200
Subject: [PATCH] doc: dd: discourage use of very large block sizes
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* doc/coreutils.texi (dd invocation): Warn against using a very large
block size. Suggested by Imre Péntek.
---
 doc/coreutils.texi | 5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)

diff --git a/doc/coreutils.texi b/doc/coreutils.texi
index 26b4eba..102ceaf 100644
--- a/doc/coreutils.texi
+++ b/doc/coreutils.texi
@@ -8020,6 +8020,11 @@ dd invocation
 @samp{w}=2, @samp{x@var{m}}=@var{m}, or any of the
 standard block size suffixes like @samp{k}=1024 (@pxref{Block size}).

+Any block size you specify via @samp{bs=}, @samp{ibs=}, @samp{obs=}, @samp{cbs=}
+should not be too large---values larger than a few megabytes
+are generally wasteful or (as in the gigabyte..exabyte case) downright
+counterproductive or error-inducing.
+
 Use different @command{dd} invocations to use different block sizes for
 skipping and I/O@. For example, the following shell commands copy data
 in 512 KiB blocks between a disk and a tape, but do not save or restore a
--
1.7.1.511.g2f531

[Dave Gilbert (ubuntu-treblig)]

Hi Jim,
  Just reading the patch, I'm not sure that's solving both cases - i.e.
     1) That there is a problem if your value is too large for RAM (i.e. a memory exhausted)
     2) There is something saying 'invalid number' on large values, and that seems to depend on the word
size of the host.

I think the original reporter was confused by case (2); IMHO 'Memory Exhausted' is enough diagnostics for (1).

Dave

[Jim Meyering (meyering)]

Dave Gilbert wrote:
> Just reading the patch, I'm not sure that's solving both cases - i.e.
> 1) That there is a problem if your value is too large for RAM (i.e. a memory exhausted)
> 2) There is something saying 'invalid number' on large values, and that seems to depend on the word
> size of the host.
>
> I think the original reporter was confused by case (2); IMHO 'Memory
> Exhausted' is enough diagnostics for (1).

Thanks for the feedback.
If you can demonstrate a precise problem, please report it.

[Imre Péntek (pentek-imre)]

Just for the records: I'm happy with the man page patch given in comment #8