Mirror of charm-nova-compute is failing due to "strange" timeout

[Alex Kavanagh]

The charm-nova-compute project holds the code upstream at opendev.org. The code mirror is failing.

Code import page: https://code.launchpad.net/~openstack-charmers/charm-nova-compute/+git/charm-nova-compute

Error from log:

2022-02-22 12:55:13 INFO Starting job.
2022-02-22 12:55:13 INFO Getting existing repository from hosting service.
2022-02-22 12:55:19 INFO Cloning into bare repository 'repository'...
2022-02-22 12:55:19 INFO remote: Enumerating objects: 27119, done.
2022-02-22 12:55:19 INFO remote: Counting objects: 100% (27119/27119)
2022-02-22 12:55:19 INFO remote: Counting objects: 100% (27119/27119), done.
2022-02-22 12:55:21 INFO remote: Compressing objects: 100% (11644/11644)
2022-02-22 12:55:21 INFO remote: Compressing objects: 100% (11644/11644), done.
2022-02-22 12:55:21 INFO Receiving objects: 98% (26577/27119), 14.57 MiB | 14.42 MiB/s
2022-02-22 12:55:21 INFO remote: Total 27119 (delta 16249), reused 25674 (delta 14827)
2022-02-22 12:55:21 INFO Receiving objects: 100% (27119/27119), 14.57 MiB | 14.42 MiB/s
2022-02-22 12:55:21 INFO Receiving objects: 100% (27119/27119), 17.43 MiB | 14.42 MiB/s, done.
2022-02-22 12:55:22 INFO Resolving deltas: 100% (16249/16249)
2022-02-22 12:55:22 INFO Resolving deltas: 100% (16249/16249), done.
2022-02-22 12:55:22 INFO Checking connectivity... done.
2022-02-22 12:55:22 INFO Fetching remote repository.
Traceback (most recent call last):
  File "/srv/lp-codeimport/payloads/0f80d22c76a72f9e9eb763eb46f19ce67bb48415/scripts/code-import-worker.py", line 112, in <module>
    sys.exit(script.main())
  File "/srv/lp-codeimport/payloads/0f80d22c76a72f9e9eb763eb46f19ce67bb48415/scripts/code-import-worker.py", line 107, in main
    return import_worker.run()
  File "/srv/lp-codeimport/payloads/0f80d22c76a72f9e9eb763eb46f19ce67bb48415/lib/lp/codehosting/codeimport/worker.py", line 576, in run
    return self._doImport()
  File "/srv/lp-codeimport/payloads/0f80d22c76a72f9e9eb763eb46f19ce67bb48415/lib/lp/codehosting/codeimport/worker.py", line 1188, in _doImport
    cwd="repository")
  File "/srv/lp-codeimport/payloads/0f80d22c76a72f9e9eb763eb46f19ce67bb48415/lib/lp/codehosting/codeimport/worker.py", line 1070, in _runGit
    for line in self._throttleProgress(git_process.stdout):
  File "/srv/lp-codeimport/payloads/0f80d22c76a72f9e9eb763eb46f19ce67bb48415/lib/lp/codehosting/codeimport/worker.py", line 1027, in _throttleProgress
    buffered, timeout=timeout):
  File "/srv/lp-codeimport/payloads/0f80d22c76a72f9e9eb763eb46f19ce67bb48415/lib/lp/codehosting/codeimport/worker.py", line 1046, in _throttleProgress
    line = next(wrapped_file)
KeyboardInterrupt
Import failed:
Traceback (most recent call last):
Failure: twisted.internet.error.TimeoutError: User timeout caused connection failure.

[Colin Watson (cjwatson)]

After hunting down a few red herrings, I'm currently baffled by this. My best theory is to add some more progress information to the `git fetch` that it's running at that point and try to see what's going on.

[Colin Watson (cjwatson)]

Adding progress information didn't help. I found a way to reproduce it reasonably reliably using just git, and bisected this to https://github.com/git/git/commit/eb049759fb6b739310af52ee0e13ce6cd0c86be7 which fixes the misbehaviour. (Even after that, the tip of git main still exhibits a similar hang if I force the use of protocol v0.)

However, this is definitely not something that can simply be cherry-picked to xenial's git, since none of the v2 stuff existed there. I think we may have to backport jammy's git to xenial.

[Alex Kavanagh (ajkavanagh)]

It's probably not an appropriate upgrade, but may be a useful starting point, but there is the git core ppa with the latest version for xenial: https://launchpad.net/~git-core/+archive/ubuntu/ppa/+packages?field.name_filter=&field.status_filter=published&field.series_filter=xenial

[Colin Watson (cjwatson)]

I've got suitable packaging already now - the current blocker is just making sure it works across our services, since we don't like to get into a situation where we're using entirely different backports on different services (though a certain amount of divergence is inevitable due to running different Ubuntu releases).

[Alex Kavanagh (ajkavanagh)]

Hi

Is there any progress on this, please? This is rapidly becoming critical as we are nearing release and can't build the charms in launchpad.

[Colin Watson (cjwatson)]

I've deployed a fix to lp-codeimport itself to allow it to *tolerate* new versions of git (upgrading git would previously have caused failures to push imported repositories back to Launchpad).

Following up to that, I've filed a ticket with our sysadmins to run a charm upgrade and to upgrade to the new git version in our PPA. I don't have the ticket number yet since some bit of email delivery somewhere seems to be slower than usual, but once that ticket is resolved it should fix this.

[Colin Watson (cjwatson)]

https://portal.admin.canonical.com/C149076

[Alex Kavanagh (ajkavanagh)]

Thanks very much; I'll keep an eye out for the progress of the ticket and then verify the code imports.

[Colin Watson (cjwatson)]

This should all be sorted now, and I've retried the code imports of charm-nova-compute, charm-ceph-mon, and charm-keystone which were the only openstack-charmers failures from this that I could find.