hpcups dies with signal 11 during cleanup, job remains in CUPS queue

[martin_jacobs]

My environment is based on SuSE Linux 8.1 with lots of improvements:
home made CUPS, HPLIP, ...
- CUPS 1.3.11
- HPLIP 3.9.10
compiled with gcc 3.4.6
Printer is HP PhotoSmart C7280 connected via WLAN

I've a job that dies at end of printing. hpcups terminates with signal 11. This error is is interpreted by CUPS as filter error and job remains in job queue. Job consists of 5 pages, all 5 are printed. Printing only part of job, e.g. first pages 1 to 4 and then 5 makes error disappear.

Here's a backtrace created with gdb:

#0 0x40338674 in chunk_free () from /lib/libc.so.6
#1 0x403385f6 in free () from /lib/libc.so.6
#2 0x4026f871 in operator delete(void*) () from /usr/lib/libstdc++.so.6
#3 0x4026f8cc in operator delete[](void*) () from /usr/lib/libstdc++.so.6
#4 0x080527d9 in ErnieFilter::~ErnieFilter() ()
#5 0x0805983f in Job::~Job() ()
#6 0x08049c02 in HPCupsFilter::~HPCupsFilter() ()
#7 0x402f65f1 in exit () from /lib/libc.so.6
#8 0x402e24aa in __libc_start_main () from /lib/libc.so.6

I've already tried running cups with log level debug, but got nothing relevant from that.

Job file from cups queue is saved.

[Naga Samrat Chowdary, Narla (samrat-hplip)]

Please run "hp-check -t" and post the output.

Thank you for supporting HPLIP,
Naga Samrat.

[martin_jacobs (martin-jacobs)]

Output of hp-check -t is added as attachment.

[martin_jacobs (martin-jacobs)]

Hi,

with the help of DUMA (http://duma.sourceforge.net/) I found two bugs in hpcups sources. One minor bug in Mode9.cpp line 215:

210 offset = 0;
211
212 if (seeded)
213 {
214 /* find a difference between the seed row and this row. */
215 while ((*sptr++ == *nptr++) && (offset < size) )
216 {
217 offset++;
218 }
219 sptr--;
220 nptr--;
221 }

this should be change, so that size check is evaluated first:

210 offset = 0;
211
212 if (seeded)
213 {
214 /* find a difference between the seed row and this row. */
215 while ( (offset < size) && (*sptr++ == *nptr++))
216 {
217 offset++;
218 }
219 sptr--;
220 nptr--;
221 }

second bug is in file Encapsulator.cpp, method Encapsulator::addToHeader does not care about target buffer pcl_buffer getting filled. Old code looks like this:

void Encapsulator::addToHeader(const BYTE *command_string, int length)
{
    memcpy(cur_pcl_buffer_ptr, command_string, length);
    cur_pcl_buffer_ptr += length;
}

new code is now:

void Encapsulator::addToHeader(const BYTE *command_string, int length)
{
    DRIVER_ERROR err;

    if (cur_pcl_buffer_ptr + length > (pcl_buffer + cur_pcl_buffer_size)) {
        err = sendBuffer(static_cast<const BYTE *>(pcl_buffer), (cur_pcl_buffer_ptr - pcl_buffer));
        if (err != NO_ERROR)
            return;
        cur_pcl_buffer_ptr = pcl_buffer;
    }
    memcpy(cur_pcl_buffer_ptr, command_string, length);
    cur_pcl_buffer_ptr += length;
}

With these changes, hpcups finishes under DUMA control without core dump and printing under cups succeedes without signal 11.

It's worth to have a check method addToHeader() with variable arguments as well.

BR
Martin

[martin_jacobs (martin-jacobs)]

My fix in Mode9.cpp needs a change, sorry. Correct version is this one:

        offset = 0;

        if (seeded)
        {
            sptr--;
            nptr--;
            /* find a difference between the seed row and this row. */
            while (*++sptr == *++nptr)
            {
                if (++offset < size)
                {
                    /* reached end of buffer */
                    sptr++;
                    nptr++;
                    break;
                }
            }
            /* sptr and nptr point behind last identical char/byte */
        }

I'll add patches for both changes.

[martin_jacobs (martin-jacobs)]

Patch for change in file Mode9.cpp: hpcups.Mode9_cpp.signal-11.patch

[martin_jacobs (martin-jacobs)]

Patch for change in file Encapsulator.cpp: hpcups.Encapsulator_cpp.signal-11.patch

[martin_jacobs (martin-jacobs)]

Sorry, patch upload failed. Once again: hpcups.Mode9_cpp.signal-11.patch

[martin_jacobs (martin-jacobs)]

and second patch: hpcups.Encapsulator_cpp.signal-11.patch

[Naga Samrat Chowdary, Narla (samrat-hplip)]

Can you please share the steps you have done with DUMA to figure out the issues.

Thank you for supporting HPLIP,
Naga Samrat Chowdary.

[martin_jacobs (martin-jacobs)]

Sorry for the delay.

1. Compile and install DUMA (libduma.so)

2. Compile hpcups without optimization and with debug symbols,
something like

 make CXXFLAGS="-march=i586" CPPFLAGS=-g LDFLAGS=-g

don't install that.

3. Pull failing print job file out of hpcups' spool directory,
something like

 d00518-001

4. Copy that print job file (containig postscript) into a
temporary directory.

5. Prepare a short shell script like this one to fake hpcups'
execution pipe:

# first compile and link with
# make CXXFLAGS="-march=i586" CPPFLAGS=-g LDFLAGS=-g
#
# replace paths and names as necessary:
#
ulimit -c unlimited
HPCUPS_BIN=<your compile path>/hplip-3.9.10/hpcups
[ -x ${HPCUPS_BIN} ] || HPCUPS_BIN=/usr/lib/cups/filter/hpcups
CMDLINE="510 owner titel 1 PageSize=A4"
[ -f core ] && rm core
( export PPD=/etc/cups/ppd/psdjet.ppd
  export DUMA_OUTPUT_FILE=check-hpcups.duma
  export DUMA_MEMCPY_OVERLAP=1
  export DEVICE_URI="hp:/net/Photosmart_C7200_series?ip=192.168.47.121"
  export CUPS_CACHEDIR=/var/cache/cups
  export CUPS_DATADIR=/usr/share/cups
  export CUPS_DOCROOT=/usr/share/doc/cups
  export CUPS_FONTPATH=/usr/share/cups/fonts
  export CUPS_REQUESTROOT=/var/spool/cups
  export CUPS_SERVERBIN=/usr/lib/cups
  export CUPS_SERVERROOT=/etc/cups
  export CUPS_STATEDIR=/var/run/cups
  export RIP_MAX_CACHE=50m
  export CONTENT_TYPE=application/postscript
  export FINAL_CONTENT_TYPE=printer/litfass
# don't let DUMA create file, it'll get strange mode settings --wxr-x--x
[ -f "${DUMA_OUTPUT_FILE}" ] && cat /dev/null > "${DUMA_OUTPUT_FILE}"
cat d00518-001 | /usr/lib/cups/filter/pstops $CMDLINE | \
/usr/lib/cups/filter/pstoraster $CMDLINE | tee d00518-001.raster | \
LD_PRELOAD=libduma.so ${HPCUPS_BIN} $CMDLINE > d00518-001.pcl
) 2> check-hpcups.debug
[ -f core ] && gdb ${HPCUPS_BIN} -c core

6. Run this script from inside your temporary directory. If
there is a memory related bug, DUMA will cancel operation,
write a core file and this script will run gdb (or whatever
you prefer, e.g. ddd) and show the line where DUMA catched the
bug.

7. Additional runtime operations of DUMA increase memory
consumption and slow the program down.

This way I discovered some more minor bugs. I'll post more
patches soon. Some of them are typical dereferencing bugs
during read operations and have no real impact but need to be
fixed to get to the "real" bugs.