K3B (growisofs) too many directories

[roger]

hello . . .

I have successfully used K3B for many years doing monthly FULL backups to DVDs and then BDs except for this month.

I tried to backup my system K3B 2.0.2 and received a few error messages primarily being told that K3B could not calculate the file size.

I downloaded Kubuntu 15.10 with K3B 2.0.3 and using a USB stick began K3B. Same error message.

I isolated the problem (maybe). growisofs generates an error telling me that my burn has "too many directories". The number 65535 (?) follows the message. This is the base error for all my other error messages. K3B cannot calculate the file size because not all the directories are calculated (?).

I removed /usr (biggest number of directories) and the burn was successful. I used a DVD-DL to burn /usr and again successful.

It looks like I have maxed out the limit on directories. Never thought that would happen.

K3b
Version 2.0.2 / 2.0.3
Using KDE Development Platform 4.14.13 and Kubuntu 15.10 downloaded 6Dec15.

Hope this helps.

[Thomas Schmitt (scdbackup)]

Hi,

the error message rather stems from genisoimage (or mkisofs)
which works underneath K3B and also underneath growisofs if
used like in the man page example

  growisofs -Z /dev/dvd -R -J /some/files

The number 65535 is a limit of the "Path Table", an antique
structure which records all directories and which hardly any
reader interprets.
Nevertheless it restricts the number of parent directories
to the range of 0 to 65535, because any entry in this table has
to refer to its boss by a 16 bit number.

The number of directories is allowed to exceed this range.
But one would have to sort apart those directories which have
directories as children (i.e. which must be mentioned as parents
in the table).

I had a look into libisofs, where i am currently the developer.
The path table code is inherited from my predecessor. It does
not take care to keep the 16 bit numbers in path table from
rolling over. (Maybe it should do ...)

So you may try to use xorrisofs out of package xorriso.
K3B might be unwilling to use it, if you do not name it "mkisofs"
or "genisoimage".

Or you may try GUI program Xfburn instead of K3B.

Have a nice day :)

Thomas

[roger (oc3vp7w8)]

Thank You for the speedy reply. I appreciate it.

Bottom line is nothing I tried that you suggested worked.

I tried xfburn prior to this bug report. The available disk sizes does not include BD. It stops at DVD-DL or my latest copy from Ubuntu stops at DVD-DL.

I first renamed genisoimage and copied xorriso to its name. The error generated was 'unable to calculate file size'.

I next renamed mkisofs and copied xorriso to its name. The error generated was 'unable to calculate file size'.

I am including both log files from my misadventures for your attention. The first is a normal report. The second is after renaming xorriso > mkisofs.

Devices
-----------------------
HL-DT-ST BD-RE BH16NS40 1.01 (/dev/sr0, CD-R, CD-RW, CD-ROM, DVD-ROM, DVD-R, DVD-RW, DVD-R DL, BD-ROM, BD-R, BD-RE, DVD+R, DVD+RW, DVD+R DL) [DVD-ROM, DVD-R Sequential, DVD-R Dual Layer Sequential, DVD-R Dual Layer Jump, DVD-RAM, DVD-RW Restricted Overwrite, DVD-RW Sequential, DVD+RW, DVD+R, DVD+R Dual Layer, CD-ROM, CD-R, CD-RW, BD-ROM, BD-R Sequential (SRM), BD-R Random (RRM), BD-RE] [SAO, TAO, RAW, SAO/R96P, SAO/R96R, RAW/R16, RAW/R96P, RAW/R96R, Restricted Overwrite, Layer Jump, Random Recording, Sequential Recording, Sequential Recording + POW] [%7]

K3b::IsoImager
-----------------------
mkisofs print size result: 0 (0 bytes)

System
-----------------------
K3b Version: 2.0.2
KDE Version: 4.14.13
QT Version: 4.8.6
Kernel: 3.16.0-55-generic

Used versions
-----------------------
mkisofs: 3.1a28

mkisofs
-----------------------
mkisofs: Warning: Cannot add inode hints with -no-cache-inodes.
mkisofs: Unable to generate sane path tables - too many directories (65579)
mkisofs: Try to use the option -no-limit-pathtables

mkisofs calculate size command:
-----------------------
/usr/bin/mkisofs -gui -graft-points -print-size -quiet -volid K3b data project -volset -appid K3B THE CD KREATOR (C) 1998-2010 SEBASTIAN TRUEG AND MICHAL MALEK -publisher -preparer -sysid LINUX -volset-size 1 -volset-seqno 1 -sort /tmp/kde-root/k3bg25874.tmp -rational-rock -hide-list /tmp/kde-root/k3bk25874.tmp -no-cache-inodes -full-iso9660-filenames -disable-deep-relocation -iso-level 3 -path-list /tmp/kde-root/k3bi25874.tmp

Devices
-----------------------
HL-DT-ST BD-RE BH16NS40 1.01 (/dev/sr0, CD-R, CD-RW, CD-ROM, DVD-ROM, DVD-R, DVD-RW, DVD-R DL, BD-ROM, BD-R, BD-RE, DVD+R, DVD+RW, DVD+R DL) [DVD-ROM, DVD-R Sequential, DVD-R Dual Layer Sequential, DVD-R Dual Layer Jump, DVD-RAM, DVD-RW Restricted Overwrite, DVD-RW Sequential, DVD+RW, DVD+R, DVD+R Dual Layer, CD-ROM, CD-R, CD-RW, BD-ROM, BD-R Sequential (SRM), BD-R Random (RRM), BD-RE] [SAO, TAO, RAW, SAO/R96P, SAO/R96R, RAW/R16, RAW/R96P, RAW/R96R, Restricted Overwrite, Layer Jump, Random Recording, Sequential Recording, Sequential Recording + POW] [%7]

K3b::IsoImager
-----------------------
mkisofs print size result: 0 (0 bytes)

System
-----------------------
K3b Version: 2.0.2
KDE Version: 4.14.13
QT Version: 4.8.6
Kernel: 3.16.0-55-generic

Used versions
-----------------------
mkisofs: 2.1-Emulation

mkisofs
-----------------------
xorriso 1.3.2 : RockRidge filesystem manipulator, libburnia project.
xorriso : FAILURE : Cannot...

[Thomas Schmitt (scdbackup)]

Hi,

> I tried xfburn prior to this bug report.

Then it must be older than version 0.5.2.
Try to get 0.5.2 or 0.5.4.

> I first renamed genisoimage and copied xorriso to its name. The error
> generated was 'unable to calculate file size'.
> ...
> xorriso : FAILURE : Cannot determine attributes of source file
> '/storage00/151206-Kubuntu64/etc/ssl/certs/NetLock_Arany_\=Class_Gold\=_Főta
> núsítvány.pem' : No such file or directory
> xorriso : FAILURE : Aborted reading of file '/tmp/kde-root/k3bK26520.tmp' in line number 3223

This is probably a bug in xorriso's mkisofs emulation.
I assume that the file name contains '=' characters which K3B
tries to hide from the -graft-points feature of mkisofs.
Therefore it inserts '\' characters before the '='.
mkisofs will remove them. Obviously xorriso does not.

"/tmp/kde-root/k3bK26520.tmp" is probably the input file for
mkisofs option -path-list. A zillion of file names in it.

Now i have to find out whether xorriso is unaware of this
gesture or whether i broke it in the past it without noticing.

If you want to give the command line a try, then we could
compose a xorriso run which would probably work.
Something like

  xorriso -for_backup -outdev /dev/sr0 -blank as_needed -add /bin /boot /etc /home /initrd.img /lib* /opt /root /sbin /usr /vmlinuz --

(I tested this with "-outdev stdio:/dev/null". The fastest
 burner ever. Up to 170x DVD speed. Regrettably it immediately
 forgets what is written to it.)

For repeated use, one would put this into a shell script so that it
can be executed by a single command word. One would write is a bit
more readable than in a single line:

  xorriso \
    -for_backup \
    -outdev /dev/sr0 \
    -blank as_needed \
    -add /bin /boot /etc /home /initrd.img /lib* /opt \
         /root /sbin /usr /vmlinuz --

(If you are not keen with shell scripting then i could give you
 a short intro. It's easy.)

One could also augment the script to do incremental backups with
several update runs to the same BD medium:
  "-dev /dev/sr0" instead of "-outdev /dev/sr0 -blank as_needed"
  "-update_r /bin /bin -update_r /boot /boot ..." instead of
  "-add ... --"

Have a nice day :)

Thomas

[Thomas Schmitt (scdbackup)]

Hi,

the problem with "\=" is embarrasingly an omission of mine.
The mkisofs emulator of xorriso inspects file addresses
("pathspecs") for this gesture only if given as program arguments,
but not if given by mkisofs option -path-list.

I will now develop a solution and make some comparisons to
the behavior of mkisofs.
Then i might bother you with:
- uninstalling Ubuntu's xorriso
- downloading xorriso-1.4.3.tar.gz
- packing that file and running: ./configure && make
- foist the resulting file ./xorriso/xorriso to K3B as "mkisofs"

Would you be willing to do this ?
(With a longer description of the procedure, of course.)

Do you have packages "gcc", "libc-dev" and "make" installed ?
(They are for C program development. You get a sufficient
 equipment for C by installing package "build-essential".)

Have a nice day :)

Thomas

[roger (oc3vp7w8)]

No problem. Glad to help.

A few words of caution. While I am very familiar with the theory behind making an executable, I have only done it in practice two or three times over the last several years and that was following written instructions in the README file.

I am now back at my machine so I can give you my following versions

xfburn 0.5.0
xorriso 1.3.2

Hope this helps.

[Thomas Schmitt (scdbackup)]

Hi,

i will make it README-grade and answer questions. :))
Only assuming that you know how to execute shell command lines
and install package "build-essential".
("valgrind" and "gdb" if we run into deep riddles.)

Meanwhile i embarassingly discovered that there are two
deviations from mkisofs habits with the interpretation of
such addresses.
My interpreter stops removing '\' after it found the
first '=' which is not preceeded by '\'.

This one affects program arguments too. It is based on a false
theory which i developed 15 years ago while i still was user
of mkisofs and cdrecord.
Nasty pitfall is that my false theory works well with mkisofs.

So: Many thanks to you and K3B for pointing these bugs out.

I am now testing a candidate for a fix.
If all goes well, i will upload a new xorriso-1.4.3.tar.gz
tomorrow. (It's the development tarball between releases.
Content may change without much notice.)

> xfburn 0.5.0

This one does not know that libburn would just burn BD as
if they were CD or DVD. Just a bit larger.

> xorriso 1.3.2

So the libburn underneath must at least be version 1.3.2, too.
It can burn BD. You'd only have to manage to upgrade xfburn and all
its many dependencies.
(libburn does BD since version 0.6.0. Nearly 7 years ago.
 How time goes by ...)

If you want to try the proposed xorriso command line, then
it is supposed to work (as superuser, of course, for the
read permissions).

Have a nice day :)

Thomas

[Thomas Schmitt (scdbackup)]

Hi,

let's try whether all fleas have been caught.
Here is the procedure:

Create a new directory (for easy removal) and go into it:

  mkdir $HOME/xorriso_dir
  cd $HOME/xorriso_dir

Download the new GNU xorriso development tarball and put it
into that new directory. E.g. by some web browser or by this
command line:

  wget http://www.gnu.org/software/xorriso/xorriso-1.4.3.tar.gz

Expect 2705623 bytes.
Program md5sum should yield 0aa887545eef544a42bef704cc7f3066.
Content and checksum may change without notice. Ask, if in doubt.

Unpack the tarball, go into the emerged directory, and let
"configure" and "make" do their magic:

  tar xzf xorriso-1.4.3.tar.gz
  cd xorriso-1.4.3
  ./configure && make

I guess ./configure will raise protest if not enough C compiler
equipment is present. As said, "build-essential" should suffice.

For the next half minute or so you should see messages hopping
up the terminal.
If no error message is to see at the end, try whether there is
a runnable program:

  xorriso/xorriso -version

which should say that it is

  GNU xorriso 1.4.3 : RockRidge filesystem manipulator, libburnia project.

plus a few lines. Among them

  xorriso version : 1.4.3
  Version timestamp : 2015.12.09.162402
  libisofs in use : 1.4.3 (min. 1.4.3)
  libburn in use : 1.4.3 (min. 1.4.3)
  libisoburn in use : 1.4.3 (min. 1.4.3)

The libraries which it reports were brought by the tarball and
are statically linked into the program. They have not been installed
system wide. There you still have versions 1.3.2.

You may reduce the program size from 5 MB to 1.5 MB (on amd64) by

  strip xorriso/xorriso

The new program

  $HOME/xorriso_dir/xorriso-1.4.3/xorriso/xorriso

may be copied or moved to any place.
If it gets started under the name "mkisofs", "genisoimage",
or "xorrisofs", then it interprets its arguments like mkisofs.
(At least i hope so ... cough ...)

Have a nice day :)

Thomas

[roger (oc3vp7w8)]

Your solution worked perfectly. I was able to burn the previous 'too many directories' with no problems. Your instructions were appreciated. Thank You.

unfortunately,

When I tried to complete the remaining full system burn, I received the following error log,

Devices
-----------------------
HL-DT-ST BD-RE BH16NS40 1.01 (/dev/sr0, CD-R, CD-RW, CD-ROM, DVD-ROM, DVD-R, DVD-RW, DVD-R DL, BD-ROM, BD-R, BD-RE, DVD+R, DVD+RW, DVD+R DL) [DVD-ROM, DVD-R Sequential, DVD-R Dual Layer Sequential, DVD-R Dual Layer Jump, DVD-RAM, DVD-RW Restricted Overwrite, DVD-RW Sequential, DVD+RW, DVD+R, DVD+R Dual Layer, CD-ROM, CD-R, CD-RW, BD-ROM, BD-R Sequential (SRM), BD-R Random (RRM), BD-RE] [SAO, TAO, RAW, SAO/R96P, SAO/R96R, RAW/R16, RAW/R96P, RAW/R96R, Restricted Overwrite, Layer Jump, Random Recording, Sequential Recording, Sequential Recording + POW] [%7]

K3b::IsoImager
-----------------------
mkisofs print size result: 0 (0 bytes)

System
-----------------------
K3b Version: 2.0.2
KDE Version: 4.14.13
QT Version: 4.8.6
Kernel: 3.16.0-55-generic

Used versions
-----------------------
mkisofs: 2.1-Emulation

mkisofs
-----------------------
GNU xorriso 1.4.3 : RockRidge filesystem manipulator, libburnia project.
xorriso : FAILURE : -as genisofs: Unsupported option '-udf'

mkisofs calculate size command:
-----------------------
/usr/bin/mkisofs -gui -graft-points -print-size -quiet -volid 2015-12-06-img -volset -appid K3B THE CD KREATOR (C) 1998-2010 SEBASTIAN TRUEG AND MICHAL MALEK -publisher -preparer -sysid LINUX -volset-size 1 -volset-seqno 1 -sort /tmp/kde-root/k3bL18829.tmp -rational-rock -hide-list /tmp/kde-root/k3by18829.tmp -no-cache-inodes -udf -full-iso9660-filenames -iso-level 3 -path-list /tmp/kde-root/k3bD18829.tmp

It appears that xorriso does not support -udt.

Thank You

Hope this helps.

[Thomas Schmitt (scdbackup)]

Hi,

> xorriso : FAILURE : -as genisofs: Unsupported option '-udf'
> It appears that xorriso does not support -udt.

Indeed, xorriso does ISO 9660, not UDF.

Ain't there any way to disable this feature in K3B ?

Two possible remedies by xorriso come to me:
--------------------------------------------------------------

You could let xorriso hop over that failure if you can talk
K3B into running

  xorriso -abort_on fatal -as mkisofs ...K3B.mkisofs.arguments...

You will still see the error message but xorriso will go on
with processing the other arguments.

K3B might complain about the exit value of the xorriso run.
You could force exit 0 (i.e. success indication) by:

  xorriso -abort_on fatal -return_with all 0 -as mkisofs ...

--------------------------------------------------------------

Second alternative is to move the software development adventure
one step further:

- Edit in directory xorriso-1.4.3 the file

    xorriso/emulators.c

  In line 711, change this

       "-no-iso-translate",

  to

       "-no-iso-translate", "-udf",

  The trailing comma is important.
  This will then be the last entry in the list ignored_arg0_options
  which as its name suggests will cause "-udf" to be ignored.
    xorriso : NOTE : -as mkisofs: Ignored option '-udf'

- After editing execute the "make" command to get a new executable
  program xorriso/xorriso

    make

--------------------------------------------------------------

All other options in the "mkisofs calculate size command"
are either supported or ignored by xorriso -as mkisofs.

Have a nice day :)

Thomas