imports hang from time to time when talking to the mass importer

[Robert Collins]

Individual import seem to hang.

1000 29087 0.0 0.0 142920 25104 ? S May21 0:00 | \_ /usr/bin/python /srv/package-import.canonical.com/new/scripts/import_package.py python-unit
strace -p 29087
Process 29087 attached - interrupt to quit
write(2, " fixed=\"getA"..., 20480 <unfinished ...>
pkg_import@jubany:/srv/package-import.canonical.com/scripts$ lsof | grep 29087.*2w
import_pa 29087 pkg_import 2w FIFO 0,5 124349124 pipe

[James Westby (james-w)]

On Fri, 04 Jun 2010 05:22:18 -0000, Robert Collins <email address hidden> wrote:
> Public bug reported:
>
> Individual import seem to hang.
>
> 1000 29087 0.0 0.0 142920 25104 ? S May21 0:00 | \_ /usr/bin/python /srv/package-import.canonical.com/new/scripts/import_package.py python-unit
> strace -p 29087
> Process 29087 attached - interrupt to quit
> write(2, " fixed=\"getA"..., 20480 <unfinished ...>
> pkg_import@jubany:/srv/package-import.canonical.com/scripts$ lsof | grep 29087.*2w
> import_pa 29087 pkg_import 2w FIFO 0,5 124349124 pipe

Probably a race in the handling of the child dying.

What I'm not sure about is what it is writing there, part of a
traceback?

Thanks,

James

[Robert Collins (lifeless)]

If we get rid of the wrapping:
                     fixed="getA

it does look like an indented line of python code doesn't it.

It could be a hung process throwing a backtrace when the tcp session
finally errors out, or something.

Not that it really matters, this is after all about a bug in the parent.

-Rob

[James Westby (james-w)]

We think this is due to this code:

        proc = subprocess.Popen("import_package.py %s" % self.package,
            shell=True, stdout=subprocess.PIPE,
            stderr=subprocess.STDOUT, stdin=subprocess.PIPE,
            preexec_fn=subprocess_setup)
        proc.stdin.close()
        while proc.poll() is None:
            if self.should_stop.isSet():
                os.kill(proc.pid, signal.SIGTERM)
            time.sleep(1)
        output = proc.stdout.read()

if the subprocess writes lots of output then it will buffer and hang, and we won't consume
until it ends, causing a deadlock.

We can't simply read() within the loop, as we don't want to block, so using select() is
probably the quick fix for this.

It's another plus for using LP's job system/ampoule/celery/something though.

Thanks,

James