"git ubuntu import" sometimes hangs on the push stage

[Robie Basak]

Running:

    git ubuntu import -v -d ... <package>

sometimes hangs at:

07/25/2017 17:27:07 - DEBUG:Executing: git push --atomic pkg "refs/heads/importer/*:refs/heads/*" "refs/tags/importer/*:refs/tags/*"

At this point, "ps axf" shows me:

20600 pts/6 S+ 0:03 | | \_ /usr/bin/python3 /home/robie/Code/Canonica/usd-importer/bin/git-ubuntu import -v -d libapache2-mod-auth-pgsql libapache2-mod-auth-pgsql
23117 pts/6 Z+ 0:00 | | \_ [git] <defunct>

After instrumenting the code to give me a full backtrace, I see:

^CTraceback (most recent call last):
  File "/home/robie/Code/Canonical/usd-importer/bin/git-ubuntu", line 18, in <module>
    main()
  File "/home/robie/Code/Canonical/usd-importer/gitubuntu/__main__.py", line 210, in main
    args.func(args)
  File "/home/robie/Code/Canonical/usd-importer/gitubuntu/importer.py", line 1273, in main
    'refs/tags/%s/*:refs/tags/*' % self.namespace,
  File "/home/robie/Code/Canonical/usd-importer/gitubuntu/git_repository.py", line 420, in git_run
    return run(['git'] + args, env=combined_env, **kwargs)
  File "/home/robie/Code/Canonical/usd-importer/gitubuntu/run.py", line 52, in run
    stdout=stdout, stderr=stderr, stdin=stdin)
  File "/usr/lib/python3.5/subprocess.py", line 695, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.5/subprocess.py", line 1072, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.5/subprocess.py", line 1715, in _communicate
    ready = selector.select(timeout)
  File "/usr/lib/python3.5/selectors.py", line 376, in select
    fd_event_list = self._poll.poll(timeout)
KeyboardInterrupt
07/25/2017 17:27:40 - INFO:Leaving /tmp/import/libapache2-mod-auth-pgsql as directed
robie@mal:/tmp/import$

This is on Xenial, currently against 43c427175132a0c66e9525ba1917a36723e029bb, but it has been happening for a while.

[Robie Basak (racb)]

Happened again with 35ffa9f6826f20797e3354caee03384f01e2b573

[Robie Basak (racb)]

This happened again when using the snap on a different machine:

installed: 0.2.2+git11.9fa9149 (291) 110MB classic

The host is running Artful.

The things in common I can think of are:

1) My Internet connection
2) My Launchpad credentials for the push

However since the hang is in Python waiting for a process, it seems unlikely to be either of those.

Timing doesn't seem to be consistent either. In my case right now, it took a while to push as it was the first import of a package.

Could it be a bug in Python in Xenial? Before Nish was running (I think) the importer on Artful? Now that it's based on Xenial, maybe Nish will see this hang sooner or later as well?

[Robie Basak (racb)]

Now that we have implemented the systemd watchdog, I think it's OK to downgrade this to Low, as it no longer impacting widespread availability.

It is still an issue because:

1) If an import of a particular source package hangs such that the watchdog fires, then it will be marked errored and won't be retried until the next upload of that source package.

2) There is still a delay before the watchdog fires, delaying the availability of other imports, but only by hours.