yum urlgrabber error in lxc-create -t fedora

[tlc]

[Impact]

 * Users wishing to use Yum on Ubuntu to build Fedora packages will be met with a backtrace due to curl changing its' API. This makes urlgrabber basically unusable without forcing turning off SSL certification verification, which is not acceptible.

[Test Case]

 * Install yum-utils
 * Ensure sslverify in section "main" is not set, or is "1" in /etc/yum/yum.conf
 * Create a working directory for testing, reference as WORKING
 * mkdir $WORKING/yum.repos.d
 * Add attached "fedora.repo" to $WORKING/yum.repos.d
 * mkdir $WORKING/var/lib/rpm
 * rpm --rootdir=$WORKING initdb
 * yumdownloader --releasever=21 --setopt-reposdir=$WORKING/yum.repos.d --destdir=$WORKING

The yumdownloader step will fail with a traceback ending in "pycurl.error: (43, 'CURLOPT_SSL_VERIFYHOST no longer supports 1 as value!')"

[Regression Potential]

 * Since the library is mostly inoperable when running without SSL verification, regressions seem unlikely.

 * The additional fix for keepalives which was included upstream is low risk also as it simply makes the code resilient to a known failure by not allowing incompatible options to be used together.

[Other Info]
N/A

---- Original Report ----

On ubuntu 13.04 with
yum: 3.4.3-1ubuntu1
lxc: 0.9.0-0ubuntu3.3
lxc-templates: 0.9.0-0ubuntu3.3

When attempting to create a fedora Linux Container, I see

$ sudo lxc-create -n fed1 -t fedora

lxc-create: No config file specified, using the default config /etc/lxc/default.conf
This is not a fedora host and release missing, defaulting to 14. use -R|--release to specify release
Checking cache download in /var/cache/lxc/fedora/x86_64/14/rootfs ...
Downloading fedora minimal ...
Fetching from http://kdeforge.unl.edu/mirrors/fedora-archive/fedora/linux/releases/14/Everything/x86_64/os//Packages/fedora-release-14-1.noarch.rpm
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 24404 100 24404 0 0 41435 0 --:--:-- --:--:-- --:--:-- 43192
rpm: RPM should not be used directly install RPM packages, use Alien instead!
rpm: However assuming you know what you are doing...
warning: /var/cache/lxc/fedora/x86_64/14/partial/fedora-release-14.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 97a1071f: NOKEY
Preparing... ################################# [100%]
 package fedora-release-14-1.noarch is already installed
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 288, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 140, in main
    result, resultmsgs = base.doCommands()
  File "/usr/share/yum-cli/cli.py", line 436, in doCommands
    self._getTs(needTsRemove)
  File "/usr/lib/python2.7/dist-packages/yum/depsolve.py", line 101, in _getTs
    self._getTsInfo(remove_only)
  File "/usr/lib/python2.7/dist-packages/yum/depsolve.py", line 112, in _getTsInfo
    pkgSack = self.pkgSack
  File "/usr/lib/python2.7/dist-packages/yum/__init__.py", line 892, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.7/dist-packages/yum/__init__.py", line 673, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.7/dist-packages/yum/repos.py", line 294, in populateSack
    sack.populate(repo, mdtype, callback, cacheonly)
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 168, in populate
    if self._check_db_version(repo, mydbtype):
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 226, in _check_db_version
    return repo._check_db_version(mdtype)
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 1268, in _check_db_version
    repoXML = self.repoXML
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 1467, in <lambda>
    repoXML = property(fget=lambda self: self._getRepoXML(),
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 1459, in _getRepoXML
    self._loadRepoXML(text=self)
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 1449, in _loadRepoXML
    return self._groupLoadRepoXML(text, self._mdpolicy2mdtypes())
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 1424, in _groupLoadRepoXML
    if self._commonLoadRepoXML(text):
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 1242, in _commonLoadRepoXML
    result = self._getFileRepoXML(local, text)
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 1020, in _getFileRepoXML
    size=102400) # setting max size as 100K
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 835, in _getFile
    result = self.grab.urlgrab(misc.to_utf8(relative), local,
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 531, in <lambda>
    grab = property(lambda self: self._getgrab())
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 526, in _getgrab
    self._setupGrab()
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 496, in _setupGrab
    self._grab = mgclass(self._grabfunc, self.urls,
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 711, in <lambda>
    urls = property(fget=lambda self: self._geturls(),
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 708, in _geturls
    self._baseurlSetup()
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 654, in _baseurlSetup
    mirrorurls.extend(list(self.metalink_data.urls()))
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 751, in <lambda>
    metalink_data = property(fget=lambda self: self._getMetalink(),
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 724, in _getMetalink
    result = ug.urlgrab(url, local, text=self.id + "/metalink")
  File "/usr/lib/python2.7/dist-packages/urlgrabber/grabber.py", line 982, in urlgrab
    return self._retry(opts, retryfunc, url, filename)
  File "/usr/lib/python2.7/dist-packages/urlgrabber/grabber.py", line 886, in _retry
    r = apply(func, (opts,) + args, {})
  File "/usr/lib/python2.7/dist-packages/urlgrabber/grabber.py", line 968, in retryfunc
    fo = PyCurlFileObject(url, filename, opts)
  File "/usr/lib/python2.7/dist-packages/urlgrabber/grabber.py", line 1063, in __init__
    self._do_open()
  File "/usr/lib/python2.7/dist-packages/urlgrabber/grabber.py", line 1350, in _do_open
    self._set_opts()
  File "/usr/lib/python2.7/dist-packages/urlgrabber/grabber.py", line 1193, in _set_opts
    self.curl_obj.setopt(pycurl.SSL_VERIFYHOST, opts.ssl_verify_host)
pycurl.error: (43, '')
Failed to download the rootfs, aborting.
Failed to download 'fedora base'
failed to install fedora
lxc-create: failed to execute template 'fedora'
lxc-create: aborted

This appears to be a yum error. /usr/share/lxc/templates/lxc-fedora is calling

  yum --installroot /var/cache/lxc/fedora/x86_64/14/partial -y --nogpgcheck install yum initscripts passwd rsyslog vim-minimal dhclient chkconfig rootfiles policycoreutils fedora-release

[tlc (tlc)]

Works on ubuntu 12.04 with
yum: 3.2.25-1ubuntu2
lxc: 0.7.5-0ubuntu67

[Jeremy Fishman (jeremy-r-fishman)]

I am seeing this same issue via 'yumdownloader'.

It appears to be a bug in urlgrabber, fixed here http://yum.baseurl.org/gitweb?p=urlgrabber.git;a=commitdiff;h=b6bfd59c4393ea8ba5165aabb249c37d6d2a9931

For explanation, see http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTSSLVERIFYHOST

The bug is still present in the latest upstream 3.9.1 release of urlgrabber.

It is possible to work-around the urlgrabber bug by setting the options 'ssl_verify_host' to 2 instead of True, though 'True' is expected value according to the pydoc:

> self.ssl_verify_host = True
>
> Check the server's hostname to make sure it matches the certificate DN

[Jeremy Fishman (jeremy-r-fishman)]

Sorry, my details:
yum: 3.4.3-1ubuntu1
yum-utils: 1.1.26-0ubuntu1
python-urlgrabber: 3.9.1-4ubuntu2

[Mike Miller (mtmiller)]

Confirmed here as well, and I can also confirm that the patch does fix the problem reported here.

[Mike Miller (mtmiller)]

Linked Debian bug #715416 reporting the same bug in urlgrabber.

[GP Solutions (gpsolutions)]

As workaround you can set sslverify to 0 in /etc/yum/yum.conf.

[Clint Byrum (clint-fewbar)]

This is pretty badly broken, and fixed already in Debian and everywhere else.

[Launchpad Janitor (janitor)]

This bug was fixed in the package urlgrabber - 3.9.1-4ubuntu5

---------------
urlgrabber (3.9.1-4ubuntu5) vivid; urgency=medium

  * Apply patch to fix urlgrabber for newer libcurl (LP: #1200288)
 -- Clint Byrum <email address hidden> Tue, 31 Mar 2015 14:34:19 -0700

[Chris J Arges (arges)]

Hello tlc, or anyone else affected,

Accepted urlgrabber into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/urlgrabber/3.9.1-4ubuntu3.14.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Clint Byrum (clint-fewbar)]

Tested on 14.04 from trusty-proposed, works!

[Clint Byrum (clint-fewbar)]

This has been verified for almost 2 weeks. Any reason it isn't released to trusty-updates yet?

[Launchpad Janitor (janitor)]

This bug was fixed in the package urlgrabber - 3.9.1-4ubuntu3.14.04.1

---------------
urlgrabber (3.9.1-4ubuntu3.14.04.1) trusty-proposed; urgency=medium

  * Apply patch to fix urlgrabber for newer libcurl (LP: #1200288)
 -- Clint Byrum <email address hidden> Tue, 31 Mar 2015 14:34:19 -0700

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for urlgrabber has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.