yum urlgrabber error in lxc-create -t fedora

Bug #1200288 reported by tlc
38
This bug affects 7 people
Affects Status Importance Assigned to Milestone
urlgrabber (Debian)
Fix Released
Unknown
urlgrabber (Ubuntu)
Fix Released
High
Unassigned
Trusty
Fix Released
High
Clint Byrum

Bug Description

[Impact]

 * Users wishing to use Yum on Ubuntu to build Fedora packages will be met with a backtrace due to curl changing its' API. This makes urlgrabber basically unusable without forcing turning off SSL certification verification, which is not acceptible.

[Test Case]

 * Install yum-utils
 * Ensure sslverify in section "main" is not set, or is "1" in /etc/yum/yum.conf
 * Create a working directory for testing, reference as WORKING
 * mkdir $WORKING/yum.repos.d
 * Add attached "fedora.repo" to $WORKING/yum.repos.d
 * mkdir $WORKING/var/lib/rpm
 * rpm --rootdir=$WORKING initdb
 * yumdownloader --releasever=21 --setopt-reposdir=$WORKING/yum.repos.d --destdir=$WORKING

The yumdownloader step will fail with a traceback ending in "pycurl.error: (43, 'CURLOPT_SSL_VERIFYHOST no longer supports 1 as value!')"

[Regression Potential]

 * Since the library is mostly inoperable when running without SSL verification, regressions seem unlikely.

 * The additional fix for keepalives which was included upstream is low risk also as it simply makes the code resilient to a known failure by not allowing incompatible options to be used together.

[Other Info]
N/A

---- Original Report ----

On ubuntu 13.04 with
yum: 3.4.3-1ubuntu1
lxc: 0.9.0-0ubuntu3.3
lxc-templates: 0.9.0-0ubuntu3.3

When attempting to create a fedora Linux Container, I see

$ sudo lxc-create -n fed1 -t fedora

lxc-create: No config file specified, using the default config /etc/lxc/default.conf
This is not a fedora host and release missing, defaulting to 14. use -R|--release to specify release
Checking cache download in /var/cache/lxc/fedora/x86_64/14/rootfs ...
Downloading fedora minimal ...
Fetching from http://kdeforge.unl.edu/mirrors/fedora-archive/fedora/linux/releases/14/Everything/x86_64/os//Packages/fedora-release-14-1.noarch.rpm
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 24404 100 24404 0 0 41435 0 --:--:-- --:--:-- --:--:-- 43192
rpm: RPM should not be used directly install RPM packages, use Alien instead!
rpm: However assuming you know what you are doing...
warning: /var/cache/lxc/fedora/x86_64/14/partial/fedora-release-14.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 97a1071f: NOKEY
Preparing... ################################# [100%]
 package fedora-release-14-1.noarch is already installed
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 288, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 140, in main
    result, resultmsgs = base.doCommands()
  File "/usr/share/yum-cli/cli.py", line 436, in doCommands
    self._getTs(needTsRemove)
  File "/usr/lib/python2.7/dist-packages/yum/depsolve.py", line 101, in _getTs
    self._getTsInfo(remove_only)
  File "/usr/lib/python2.7/dist-packages/yum/depsolve.py", line 112, in _getTsInfo
    pkgSack = self.pkgSack
  File "/usr/lib/python2.7/dist-packages/yum/__init__.py", line 892, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.7/dist-packages/yum/__init__.py", line 673, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.7/dist-packages/yum/repos.py", line 294, in populateSack
    sack.populate(repo, mdtype, callback, cacheonly)
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 168, in populate
    if self._check_db_version(repo, mydbtype):
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 226, in _check_db_version
    return repo._check_db_version(mdtype)
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 1268, in _check_db_version
    repoXML = self.repoXML
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 1467, in <lambda>
    repoXML = property(fget=lambda self: self._getRepoXML(),
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 1459, in _getRepoXML
    self._loadRepoXML(text=self)
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 1449, in _loadRepoXML
    return self._groupLoadRepoXML(text, self._mdpolicy2mdtypes())
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 1424, in _groupLoadRepoXML
    if self._commonLoadRepoXML(text):
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 1242, in _commonLoadRepoXML
    result = self._getFileRepoXML(local, text)
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 1020, in _getFileRepoXML
    size=102400) # setting max size as 100K
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 835, in _getFile
    result = self.grab.urlgrab(misc.to_utf8(relative), local,
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 531, in <lambda>
    grab = property(lambda self: self._getgrab())
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 526, in _getgrab
    self._setupGrab()
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 496, in _setupGrab
    self._grab = mgclass(self._grabfunc, self.urls,
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 711, in <lambda>
    urls = property(fget=lambda self: self._geturls(),
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 708, in _geturls
    self._baseurlSetup()
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 654, in _baseurlSetup
    mirrorurls.extend(list(self.metalink_data.urls()))
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 751, in <lambda>
    metalink_data = property(fget=lambda self: self._getMetalink(),
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 724, in _getMetalink
    result = ug.urlgrab(url, local, text=self.id + "/metalink")
  File "/usr/lib/python2.7/dist-packages/urlgrabber/grabber.py", line 982, in urlgrab
    return self._retry(opts, retryfunc, url, filename)
  File "/usr/lib/python2.7/dist-packages/urlgrabber/grabber.py", line 886, in _retry
    r = apply(func, (opts,) + args, {})
  File "/usr/lib/python2.7/dist-packages/urlgrabber/grabber.py", line 968, in retryfunc
    fo = PyCurlFileObject(url, filename, opts)
  File "/usr/lib/python2.7/dist-packages/urlgrabber/grabber.py", line 1063, in __init__
    self._do_open()
  File "/usr/lib/python2.7/dist-packages/urlgrabber/grabber.py", line 1350, in _do_open
    self._set_opts()
  File "/usr/lib/python2.7/dist-packages/urlgrabber/grabber.py", line 1193, in _set_opts
    self.curl_obj.setopt(pycurl.SSL_VERIFYHOST, opts.ssl_verify_host)
pycurl.error: (43, '')
Failed to download the rootfs, aborting.
Failed to download 'fedora base'
failed to install fedora
lxc-create: failed to execute template 'fedora'
lxc-create: aborted

This appears to be a yum error. /usr/share/lxc/templates/lxc-fedora is calling

  yum --installroot /var/cache/lxc/fedora/x86_64/14/partial -y --nogpgcheck install yum initscripts passwd rsyslog vim-minimal dhclient chkconfig rootfiles policycoreutils fedora-release

Revision history for this message
tlc (tlc) wrote :

Works on ubuntu 12.04 with
yum: 3.2.25-1ubuntu2
lxc: 0.7.5-0ubuntu67

Revision history for this message
Jeremy Fishman (jeremy-r-fishman) wrote :

I am seeing this same issue via 'yumdownloader'.

It appears to be a bug in urlgrabber, fixed here http://yum.baseurl.org/gitweb?p=urlgrabber.git;a=commitdiff;h=b6bfd59c4393ea8ba5165aabb249c37d6d2a9931

For explanation, see http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTSSLVERIFYHOST

The bug is still present in the latest upstream 3.9.1 release of urlgrabber.

It is possible to work-around the urlgrabber bug by setting the options 'ssl_verify_host' to 2 instead of True, though 'True' is expected value according to the pydoc:

> self.ssl_verify_host = True
>
> Check the server's hostname to make sure it matches the certificate DN

Revision history for this message
Jeremy Fishman (jeremy-r-fishman) wrote :

Sorry, my details:
yum: 3.4.3-1ubuntu1
yum-utils: 1.1.26-0ubuntu1
python-urlgrabber: 3.9.1-4ubuntu2

Revision history for this message
Mike Miller (mtmiller) wrote :

Confirmed here as well, and I can also confirm that the patch does fix the problem reported here.

affects: yum (Ubuntu) → urlgrabber (Ubuntu)
Changed in urlgrabber (Ubuntu):
status: New → Confirmed
Revision history for this message
Mike Miller (mtmiller) wrote :

Linked Debian bug #715416 reporting the same bug in urlgrabber.

Changed in urlgrabber (Debian):
status: Unknown → New
Revision history for this message
GP Solutions (gpsolutions) wrote :

As workaround you can set sslverify to 0 in /etc/yum/yum.conf.

Changed in urlgrabber (Debian):
status: New → Fix Released
Revision history for this message
Clint Byrum (clint-fewbar) wrote :

This is pretty badly broken, and fixed already in Debian and everywhere else.

Changed in urlgrabber (Ubuntu):
status: Confirmed → Triaged
importance: Undecided → Critical
importance: Critical → High
Changed in urlgrabber (Ubuntu Trusty):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Clint Byrum (clint-fewbar)
summary: - yum error in lxc-create -t fedora
+ yum urlgrabber error in lxc-create -t fedora
description: updated
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package urlgrabber - 3.9.1-4ubuntu5

---------------
urlgrabber (3.9.1-4ubuntu5) vivid; urgency=medium

  * Apply patch to fix urlgrabber for newer libcurl (LP: #1200288)
 -- Clint Byrum <email address hidden> Tue, 31 Mar 2015 14:34:19 -0700

Changed in urlgrabber (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello tlc, or anyone else affected,

Accepted urlgrabber into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/urlgrabber/3.9.1-4ubuntu3.14.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in urlgrabber (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Tested on 14.04 from trusty-proposed, works!

tags: added: verification-done
removed: verification-needed
Revision history for this message
Clint Byrum (clint-fewbar) wrote :

This has been verified for almost 2 weeks. Any reason it isn't released to trusty-updates yet?

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package urlgrabber - 3.9.1-4ubuntu3.14.04.1

---------------
urlgrabber (3.9.1-4ubuntu3.14.04.1) trusty-proposed; urgency=medium

  * Apply patch to fix urlgrabber for newer libcurl (LP: #1200288)
 -- Clint Byrum <email address hidden> Tue, 31 Mar 2015 14:34:19 -0700

Changed in urlgrabber (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Chris Halse Rogers (raof) wrote : Update Released

The verification of the Stable Release Update for urlgrabber has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.