Cuneiform fails when invoked from a script

[Christian Güdel]

I'm trying to build one-touch scanning using scanbuttond + cuneiform + hocr2pdf. This works when invoked manually, however not when invoked by scanbuttond.

Here's the script I'm using:

#!/bin/bash -x
TMPDIR=/tmp/scan/
SCANDIR=/var/spool/sane/
DEVICE=$1

# -----------------
COUNTER="`cat $SCANDIR/.counter`"
[ -z $COUNTER ] && COUNTER=1
[ ! -e $TMPDIR ] && mkdir -p $TMPDIR

/usr/bin/scanimage -d $DEVICE \
     --resolution 300 \
     --mode Color \
     --source ADF \
     --format tiff \
     --batch=$TMPDIR/out%d.tiff

sleep 5

for page in $TMPDIR/out*.tiff; do
 /opt/cuneiform/bin/cuneiform -f hocr -l ger -o $TMPDIR/ocr.hocr $page
 /usr/bin/hocr2pdf -i $page -o $SCANDIR/`date +%Y%m%d-%H%M`-$COUNTER.pdf < $TMPDIR/ocr.hocr
 COUNTER=$[ $COUNTER + 1 ];
done

# cleanup
#rm -r $TMPDIR/*

echo $COUNTER > $SCANDIR/.counter

When I invoke the script manually from the command line, everything works fine. However when scanbuttond invokes it after pressing a button on the scanner, cuneiform produces the following error:

PUMA_Init failed.
Cuneiform for Linux 0.6.0

I'm out of ideas on this one, I can even start the script with env -, so it's not some environment-related thing.

[Yury V. Zaytsev (zyv)]

"PUMA_Init failed" means that Cuneiform console client failed to initialize the recognition engine. This can happen in various circumstances, notably because:

1) It can't initialise the libraries (use LD_LIBRARY_PATH)
2) It can't read the *.dat files, containing the language support etc.

Sounds like a permission problem to me if it works when you run it manually.

[Christian Güdel (cg-dmesg-deactivatedaccount)]

I figured out something like that. I just tried to use LD_LIBRARY_PATH=/opt/cuneiform/lib64, but that gives the same error. It's very very unlikely to be a permission problem as the script currently runs as root.

Where does it read the *.dat files? Is there an environment variable that needs to be set?

[Yury V. Zaytsev (zyv)]

Normally it should be INSTALL_PREFIX/share/cuneiform. Have you tried running it as your current user, by the way?

I would recommend you trying to run it through strace so we can see exactly at which point it fails, because otherwise it's more like a guessing game.

[Christian Güdel (cg-dmesg-deactivatedaccount)]

Attached is the output of my script (#!/bin/bash -x) and strace.log

[Jussi Pakkanen (jpakkane)]

If your libraries are in the wrong place, you get a system linker error rather than the text you got. I have no idea why your system does not work from a script are you sure you installed everything with "make install"?

You can specify the directory Cuneiform uses for data with the environment variable CF_DATADIR. Try pointing that to your data dir in the script.

[Yury V. Zaytsev (zyv)]

Something I can see right away (not sure whether this might have caused the problem or not) is that you have double slashes in your filenames like /tmp/scan//out1.tiff.

Other than that, it mysteriously fails right after

ModulesInit -> RSTR_Init -> p2_msk_init -> MSKInit("rec4inc.dat")

but it seems to have read this file, so obviously it can find it somehow. Sorry, but I am not of much help here :(

I find it strange that it only fails if evoked by the daemon with root privileges. Did you try to evoke it manually using sudo as your current user and as root on the same file and compare the differences between strace logs?

[Christian Güdel (cg-dmesg-deactivatedaccount)]

I installed cuneiform via en ebuild now (so it's installed to /usr now). The problem unfortunately isn't solved :(
I also tried to set CF_DATADIR to /usr/share/cuneiform, doesn't help too.

Output:
+ TMPDIR=/tmp/scan
+ SCANDIR=/var/spool/sane
+ DEVICE=hp5590:libusb:008:003
++ cat /var/spool/sane/.counter
+ COUNTER=15
+ '[' -z 15 ']'
+ '[' '!' -e /tmp/scan ']'
+ /usr/bin/scanimage -d hp5590:libusb:008:003 --resolution 300 --mode Color --source ADF --format tiff --batch=/tmp/scan/out%d.tiff
Scanning -1 pages, incrementing by 1, numbering from 1
Scanning page 1
Scanned page 1. (scanner status = 5)
Scanning page 2
scanimage: sane_start: Document feeder out of documents
+ for page in '$TMPDIR/out*.tiff'
+ CF_DATADIR=/usr/share/cuneiform
+ /usr/bin/cuneiform -f hocr -l ger -o /tmp/scan/ocr.hocr /tmp/scan/out1.tiff
PUMA_Init failed.
Cuneiform for Linux 0.6.0

++ date +%Y%m%d-%H%M
+ /usr/bin/hocr2pdf -i /tmp/scan/out1.tiff -o /var/spool/sane/20090304-1543-15.pdf
/etc/scanbuttond/scripts/one-touch.sh: line 20: /tmp/scan/ocr.hocr: No such file or directory
+ COUNTER=16
+ echo 16

------------
Just after that I run the exact same command as the same user (root):
mojito ~ # /usr/bin/cuneiform -f hocr -l ger -o /tmp/scan/ocr.hocr /tmp/scan/out1.tiff
Cuneiform for Linux 0.6.0
The image depth is 24 at this point.

So everything works perfectly. Attached is the strace.log of the manual run.

[yaleks (yaleks)]

2009/3/4 Christian Güdel:
> I installed cuneiform via en ebuild now (so it's installed to /usr now). The problem unfortunately isn't solved :(
> I also tried to set CF_DATADIR to /usr/share/cuneiform, doesn't help too.
Where do you get this ebuild? From sunrise overlay or bugzilla?

[Christian Güdel (cg-dmesg-deactivatedaccount)]

The ebuild is from sunrise overlay.

[Christian Güdel (cg-dmesg-deactivatedaccount)]

Is there anything else you need to know to fix this issue? gdb backtrace when it fails or something else?

[yaleks (yaleks)]

2009/3/5 Christian Güdel <email address hidden>:
> The ebuild is from sunrise overlay.
Have you tried to recognize demo images, included in tar.bz2 (russian
and english)?

[Yury V. Zaytsev (zyv)]

Hello, Christian,

I am willing to help you, but unfortunately I am not as knowledgeable as the other devs and practically out of ideas. The fact that we can't reproduce it does not help either. I will try to compare the stack traces whenever I get time, but I am not sure how far we could get with this...

P.S. I have no experience with dbg traces, unfortunately.

[Christian Güdel (cg-dmesg-deactivatedaccount)]

I also tried to use a demo image (images/book_1.tif from cuneiform-0.6.0.tar.bz2), which fails with the same message when launched from scanbuttond.

Executing the same command afterwards works though (as expected).

[yaleks (yaleks)]

2009/3/6 Christian Güdel <email address hidden>:
> I also tried to use a demo image (images/book_1.tif from
> cuneiform-0.6.0.tar.bz2), which fails with the same message when
> launched from scanbuttond.
>
> Executing the same command afterwards works though (as expected).
>
Please run your script with "whoami" and "env" inside with scanbuttond
and as normal user.

[Christian Güdel (cg-dmesg-deactivatedaccount)]

Sure, here's the log when launched from scanbuttond.

[Christian Güdel (cg-dmesg-deactivatedaccount)]

And log when launched as root from console.

[yaleks (yaleks)]

2009/3/7 Christian Güdel:
> Sure, here's the log when launched from scanbuttond.
>
Run cuneiform with "strace /usr/bin/cuneiform ..." and add "ldd
/usr/bin/cuneiform" to the script.

[Yury V. Zaytsev (zyv)]

Yaleks, check out those stack traces:

http://launchpadlibrarian.net/23317027/strace.log
http://launchpadlibrarian.net/23424525/strace2.log

I've completely forgot about ldd! I should have asked this in the first place... :-(

[yaleks (yaleks)]

2009/3/8 Yury V. Zaytsev:
> Yaleks, check out those stack traces:
>
> http://launchpadlibrarian.net/23317027/strace.log
> http://launchpadlibrarian.net/23424525/strace2.log
>
> I've completely forgot about ldd! I should have asked this in the first
> place... :-(
This is from different computers?

[Yury V. Zaytsev (zyv)]

The first one is when cuneiform is evoked by scanbuttond with root privileges, the second one when it was run manually by root (see above).

[yaleks (yaleks)]

2009/3/8 Yury V. Zaytsev:
> The first one is when cuneiform is evoked by scanbuttond with root
> privileges, the second one when it was run manually by root (see above).
Look at first line:

1: execve("/opt/cuneiform/bin/cuneiform",
["/opt/cuneiform/bin/cuneiform", "-f", "hocr", "-l", "ger", "-o",
"/tmp/scan//ocr.hocr", "/tmp/scan//out1.tiff"], [/* 14 vars */]) = 0

2: execve("/usr/bin/cuneiform", ["/usr/bin/cuneiform", "-f", "hocr",
"-l", "ger", "-o", "/tmp/scan/ocr.hocr", "/tmp/scan/out1.tiff"], [/*
35 vars */]) = 0

You have several installations on one computer. One of them is
invalid. Don't install software "by hand", use package manages
(portage, rpm, apt, checkinstall, etc...)

[Christian Güdel (cg-dmesg-deactivatedaccount)]

I don't have two installations, the first one was installed manually from cuneiform-0.6.0.tar.bz2. I saw the problem there. After discovering the ebuild in the sunrise overlay, I rm'd /opt/cuneiform and reinstalled using portage.

Attached is a backtrace from the current installation and the output of my script including ldd.

[Christian Güdel (cg-dmesg-deactivatedaccount)]

And the output of the script.

[Christian Güdel (cg-dmesg-deactivatedaccount)]

And here's the same log starting the script as root (from the command-line)

[Jussi Pakkanen (jpakkane)]

There seems to be some issue with rec4cour.dat. Scanbutton does this:

open("/usr/share/cuneiform/rec4cour.dat", O_RDONLY) = 0
open("", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/cuneiform/rec4inc.dat", O_RDONLY) = 3
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 32) = 32

That is, Cuneiform opens it successfully but does not read any data from it.

Running from command line does this:

open("/usr/share/cuneiform/rec4cour.dat", O_RDONLY) = 3
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 32) = 32
read(3, "C\0"..., 2) = 2
read(3, "\0\0"..., 2) = 2
read(3, "\0\0"..., 2) = 2
read(3, "\0\0"..., 2) = 2

That is, it opens correctly and reads stuff from it. It also gets a different fd number. Don't know why.

[Christian Güdel (cg-dmesg-deactivatedaccount)]

Any news on this? Anything I can do to trace back the problem?

[Jussi Pakkanen (jpakkane)]

If you can, run gdb, break where it opens rec4cour.dat and step along to find out the point where the two executions differ. That is, why the first does not read data but the second one does.

[Christian Güdel (cg-dmesg-deactivatedaccount)]

This problem gets stranger and stranger. When running under gdb, it all works!

Here's the output. File 1 is the output of /bin/bash -x and file 2 the output of the gdb command.

[Jussi Pakkanen (jpakkane)]

Usually these symptoms mean a timing or threading error. Which should not be the case here, since CF is very strongy single-threaded. In theory the patch in bug 344965 could help, though it is improbable. Can you try it?

[Jussi Pakkanen (jpakkane)]

If you check the very first line of the logs, you see this when running as regular user:

execve("/usr/bin/cuneiform", ["/usr/bin/cuneiform", "-f", "hocr", "-l", "ger", "-o", "/tmp/scan/ocr.hocr", "/tmp/scan/out1.tiff"], [/* 13 vars */]) = 0

For root you get this:

execve("/usr/bin/cuneiform", ["/usr/bin/cuneiform", "-f", "hocr", "-l", "ger", "-o", "/tmp/scan/ocr.hocr", "/tmp/scan/out1.tiff"], [/* 34 vars */]) = 0

That is, it gets executed with a different set of parameters. I have no idea what those do. Does root get different protections or permissions? Also note line 6:

open("/etc/ld.so.cache", O_RDONLY) = 3

versus

open("/etc/ld.so.cache", O_RDONLY) = 0

Why does it get a different file descriptor?

One more thing to try is running CF as root by itself from script. Create the following script:

#!/bin/bash -x
/usr/bin/cuneiform -f hocr -l ger -o /tmp/scan/ocr.hocr /tmp/scan/out1.tiff

(CF command taken from your earlier post.)

And run it as root. This should tell us if the crash is caused by Bash or by something else.