Unable to fetch git repos via python subprocess with proxy configured git 2.17.0-1ubuntu1

[Po-Hsu Lin]

All the tests that need to fetch a git repo for testing failed with error message like:
    CmdError: Command <git clone https://github.com/tytso/xfstests-bld> failed, rc=128, Command returned non-zero exit status
    * Command:
        git clone https://github.com/tytso/xfstests-bld
    Exit status: 128
    Duration: 0.254368066788

    stderr:
    Cloning into 'xfstests-bld'...
    fatal: unable to access 'https://github.com/tytso/xfstests-bld/': gnutls_handshake() failed: An unexpected TLS packet was received.

Step to reproduce:
  1. Deploy a Bionic system and install the python-minimal package
  2. Run the following python code:
 import subprocess
 import os

 proxysets = [{'addr': 'squid.internal', 'desc': 'Running in the Canonical CI environment'},
              {'addr': '91.189.89.216', 'desc': 'Running in the Canonical enablement environment'},
              {'addr': '10.245.64.1', 'desc': 'Running in the Canonical enablement environment'}]
for proxy in proxysets:
        cmd = ['nc', '-w' ,'2', proxy['addr'], '3128']
        try:
            subprocess.check_call(cmd)
            print proxy['desc']
            os.environ['https_proxy'] = 'https://' + proxy['addr'] + ':3128'
            break
        except:
            pass

 subprocess.check_call(['git', 'clone', 'https://github.com/tytso/xfstests-bld'])
   4. Update the git package from 2.15.1-1ubuntu2 to 2.17.0-1ubuntu1
   5. Remove the xfstests-bld dir and try step 3 again

Result:
  * Step 1 - 3 with git 2.15.1 will work, but you will see the error message with git updated to 2.17.0 (this code works without proxy configured)

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: linux-image-4.15.0-15-generic 4.15.0-15.16
ProcVersionSignature: Ubuntu 4.15.0-15.16-generic 4.15.15
Uname: Linux 4.15.0-15-generic s390x
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu4
Architecture: s390x
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
CurrentDmesg:

Date: Tue Apr 10 11:35:24 2018
HibernationDevice: RESUME=UUID=e2735dd1-cbbc-4424-9c25-d37d073d9414
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1:
PciMultimedia:

ProcFB: Error: [Errno 2] No such file or directory: '/proc/fb'
ProcKernelCmdLine: root=UUID=38f7d570-2973-4055-b668-12a9ace1e0c7 crashkernel=196M BOOT_IMAGE=0
RelatedPackageVersions:
 linux-restricted-modules-4.15.0-15-generic N/A
 linux-backports-modules-4.15.0-15-generic N/A
 linux-firmware 1.173
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)

[Po-Hsu Lin (cypressyew)]

Tested with an older kernel on the same s390x node (kernel02), this issue still exists.

Must be something else.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1762709

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Po-Hsu Lin (cypressyew)]

This can be reproduced with the following python code on Bionic, with the autotest framework cloned:

git clone --depth=1 git://kernel.ubuntu.com/ubuntu/autotest
sudo apt-get install -y python-minimal
python
from autotest.client import test, utils
import os

proxysets = [{'addr': 'squid.internal', 'desc': 'Running in the Canonical CI environment'},
                  {'addr': '91.189.89.216', 'desc': 'Running in the Canonical enablement environment'},
                  {'addr': '10.245.64.1', 'desc': 'Running in the Canonical enablement environment'}]
for proxy in proxysets:
            cmd = 'nc -w 2 ' + proxy['addr'] + ' 3128'
            try:
                utils.system_output(cmd, retain_output=False)
                print proxy['desc']
                os.environ['http_proxy'] = 'http://' + proxy['addr'] + ':3128'
                os.environ['https_proxy'] = 'https://' + proxy['addr'] + ':3128'
                break
            except:
                pass

utils.system('git clone https://github.com/tytso/xfstests-bld')

But it works without the proxy setting in the python code above.
I will file a RT for this

[Po-Hsu Lin (cypressyew)]

On a second thought, since it works with proxy configured from the shell, RT ticket might not be helpful on this.

This looks like an issue in some package.

ubuntu@kernel02:/tmp$ if echo "" | nc -w 2 squid.internal 3128 >/dev/null 2>&1; then
> echo "Running in the Canonical CI environment"
> export http_proxy="http://squid.internal:3128"
> export https_proxy="http://squid.internal:3128"
> elif echo "" | nc -w 2 91.189.89.216 3128 >/dev/null 2>&1; then
> echo "Running in the Canonical enablement environment"
> export http_proxy="http://91.189.89.216:3128"
> export https_proxy="http://91.189.89.216:3128"
> elif echo "" | nc -w 2 10.245.64.1 3128 >/dev/null 2>&1; then
> echo "Running in the Canonical enablement environment"
> export http_proxy="http://10.245.64.1:3128"
> export https_proxy="http://10.245.64.1:3128"
> fi
Running in the Canonical CI environment
ubuntu@kernel02:/tmp$ echo $https_proxy
http://squid.internal:3128
ubuntu@kernel02:/tmp$ git clone https://github.com/tytso/xfstests-bld
Cloning into 'xfstests-bld'...
remote: Counting objects: 6435, done.
remote: Compressing objects: 100% (80/80), done.
remote: Total 6435 (delta 51), reused 96 (delta 43), pack-reused 6309
Receiving objects: 100% (6435/6435), 4.78 MiB | 1.66 MiB/s, done.
Resolving deltas: 100% (3943/3943), done.

[Po-Hsu Lin (cypressyew)]

This seem to be related to the git package.
I will update the description.

[Po-Hsu Lin (cypressyew)]

More tests:
  1. PASS - proxy configured with export, git clone with python
  2. FAIL - proxy configures with python, git clone with python

And I think I have found the cause:
The https_proxy should be using the proxy server with http, not https.

Not sure why this works with older git version (and why it works with proxy configured in shell), maybe the check was not that strict?

Anyway, I will prepare the fix for the SRU testing infrastructure.

[Po-Hsu Lin (cypressyew)]

Contacted IS, yes we should use http for https_proxy:

<PHLin> sajoupa, I would like to know if I set my https_proxy to it, should I use https://squid.internal:3128 or http://squid.internal:3128?
<sajoupa> PHLin: set it to http://
<PHLin> sajoupa, ok, just curious because it used to work with https, but not with git package updated
<PHLin> sajoupa, thanks
<sajoupa> PHLin: yes, it usually works with https:// but the supported way is http://
<PHLin> sajoupa, so it should not be considered as a bug if https:// is not working anymore, right?
<sajoupa> PHLin: no

[Po-Hsu Lin (cypressyew)]

Patch applied, tested with positive result, I will just close this bug with Fix-released.

http://kernel.ubuntu.com/git/ubuntu/autotest-client-tests.git/commit/?h=master-next
http://kernel.ubuntu.com/git/ubuntu/kernel-testing.git/commit/?h=master-next&id=712c14f8b9e23c2769e3e43a39d116289da61ea9