python-pip provider fails

[Simon Dodsley]

Ubuntu bug: https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1306991
Affects Fuel 8.0
Causes post-deploymrent plugin to fail when trying to do a pip install.

Following code extract causes failure:

    package {"purestorage":
      ensure => "installed",
      provider => pip
}
See: https://github.com/openstack/fuel-plugin-purestorage-cinder/blob/master/deployment_scripts/puppet/modules/plugin_purestorage_cinder/manifests/controller.pp

Error is:

2016-02-18 00:13:03 +0000 Puppet (debug): Prefetching pip resources for package
2016-02-18 00:13:03 +0000 Puppet (debug): Executing '/usr/bin/pip freeze'
2016-02-18 00:13:04 +0000 Puppet (err): Could not prefetch package provider 'pip': [nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil]
/usr/lib/ruby/vendor_ruby/puppet/util/execution.rb:86:in `execpipe'
/usr/lib/ruby/vendor_ruby/puppet/provider.rb:132:in `execpipe'
/usr/lib/ruby/vendor_ruby/puppet/provider/package/pip.rb:29:in `instances'
/usr/lib/ruby/vendor_ruby/puppet/provider/package.rb:4:in `prefetch'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:307:in `prefetch'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:197:in `prefetch_if_necessary'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:96:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:116:in `call'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:116:in `traverse'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
/usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:244:in `override'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:244:in `override'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
/usr/bin/puppet:8:in `<main>'
2016-02-18 00:13:04 +0000 /Package[purestorage] (info): Starting to evaluate the resource
2016-02-18 00:13:04 +0000 Puppet (debug): Executing '/usr/bin/pip freeze'
2016-02-18 00:13:04 +0000 /Package[purestorage] (err): Could not evaluate: [nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil]
/usr/lib/ruby/vendor_ruby/puppet/util/execution.rb:86:in `execpipe'
/usr/lib/ruby/vendor_ruby/puppet/provider.rb:132:in `execpipe'
/usr/lib/ruby/vendor_ruby/puppet/provider/package/pip.rb:29:in `instances'
/usr/lib/ruby/vendor_ruby/puppet/provider/package/pip.rb:49:in `query'
/usr/lib/ruby/vendor_ruby/puppet/provider/package.rb:19:in `properties'
/usr/lib/ruby/vendor_ruby/puppet/type/package.rb:184:in `retrieve'
/usr/lib/ruby/vendor_ruby/puppet/type.rb:1048:in `retrieve'
/usr/lib/ruby/vendor_ruby/puppet/type.rb:1076:in `retrieve_resource'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:236:in `from_resource'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:19:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:204:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:217:in `eval_resource'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark'
/usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
/usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:244:in `override'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:244:in `override'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
/usr/bin/puppet:8:in `<main>'

[Dmitry Ilyin (idv1985)]

I can't reproduce this. For me pip provider works well with pip either installed by "apt-get install python-pip" or by "easy_install pip".
Your trace is very strange and I don;t know what could have caused this.
Do you still have the environment where this situation can be reproduced?

[Simon Dodsley (simon-dodsley)]

Yes I do, but I'm away most of next week so will have limited access to it.

What do you need?

[Andrey Danin (gcon-monolake)]

Here is a diagnostic snapshot. Dmitry please take a look.

[Dmitry Ilyin (idv1985)]

I've accidentally reproduces this

Debug: Executing '/usr/bin/pip freeze'
Error: Could not prefetch package provider 'pip': [nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil]

It's because pip freeze command is failing:

> pip freeze

Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/freeze.py", line 74, in run
    req = pip.FrozenRequirement.from_dist(dist, dependency_links, find_tags=find_tags)
  File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 289, in from_dist
    assert len(specs) == 1 and specs[0][0] == '=='
AssertionError

Why does it fail?

Because in this assertion: assert len(specs) == 1 and specs[0][0] == '=='
We got spec value: [('===', '0.9.3.5ubuntu2')]

Looks like it does not know that there can be 3 equal signs.
I have no idea what do these 3 equal signs mean and why they are here.

This package is: python-pip (1.5.4-1ubuntu3)

Ok, i've removed "python-pip" deb package and used easy_install to install "pip" egg.

Here:
/usr/local/lib/python2.7/dist-packages/pip-8.1.0-py2.7.egg/pip/__init__.py
We have:
assert len(specs) == 1 and specs[0][0] in ["==", "==="],

Oh, now they know that there can be 3 equal signs. and freeze command is working.
Now the puppet provider can receive the correct command output and is working.

I can only advice not to use "python-pip" deb package. I've had some problems with it before and i'm using pip from egg instead.

How can we fix this on the fuel nodes?
It would be good to build a new "python-pip" package that actually works and install it instead of the standard Ubuntu one.

Or, if you really need a quick fix, you can do something like this:

package { 'python-pip' :
  ensure => 'absent',
}

 ->

exec { 'install-pip' :
  command => '/usr/bin/easy_install pip',
  creates => '/usr/local/bin/pip',
}

->

Package<| provider == 'pip' |>

Or just reinstall pip manually everywhere.

[Dmitry Ilyin (idv1985)]

These guys are having the same problem: http://stackoverflow.com/questions/27711184/why-is-pip-raising-an-assertionerror-on-pip-freeze

[Denis Meltsaykin (dmeltsaykin)]

Dmitry, did I understand it correctly that the issue is not in the product itself and we have nothing to fix?

[Dmitry Ilyin (idv1985)]

Pip is actually an upstream Ubuntu package and we should not be overriding it even if the package doesn't work as intended.
If you are affected by this bug just remove the 'pip' installed by apt-get and install pip using easy_install.

[Simon Dodsley (simon-dodsley)]

Given this is fixed in 9.0 as the pip provided in that version of Ubuntu doesn't have the issue I'll let you off this statement, but I actually consider this 'Won't fix' to be a lazy opt out on your part.

[Denis Meltsaykin (dmeltsaykin)]

Simon, as it was stated previously the problem is not in our code/product. This problem persists in Ubuntu 14.04 and must be fixed by their teams. Please use the workaround which has been kindly provided by Dmitry.