check_grenade_dsvm failed - aggregates tests failed

[Xing Yang]

Multiple errors in check_grenade_dsvm logs:
http://logs.openstack.org/58/64958/3/check/check-grenade-dsvm/5133512/console.html

The real failure seems to be related to "aggregates":
2014-01-04 18:15:59.936 | FAILED aggregates

I see the following errors:
2014-01-04 18:10:54.459 | 2014-01-04 18:10:54 Downloading/unpacking python-keystoneclient>=0.2.1,<0.3 (from -r keystone.egg-info/requires.txt (line 13))
2014-01-04 18:10:54.462 | 2014-01-04 18:10:54 Storing download in cache at /var/cache/pip/http%3A%2F%2Fpypi.openstack.org%2Fopenstack%2Fpython-keystoneclient%2Fpython-keystoneclient-0.2.5.tar.gz
2014-01-04 18:10:54.466 | 2014-01-04 18:10:54 Running setup.py egg_info for package python-keystoneclient
2014-01-04 18:10:54.470 | 2014-01-04 18:10:54
2014-01-04 18:10:54.474 | 2014-01-04 18:10:54 Installed /opt/stack/old/keystone/build/python-keystoneclient/d2to1-0.2.11-py2.7.egg
2014-01-04 18:10:54.478 | 2014-01-04 18:10:54 [pbr] Processing SOURCES.txt
2014-01-04 18:10:54.481 | 2014-01-04 18:10:54 warning: LocalManifestMaker: standard file '-c' not found
2014-01-04 18:10:54.485 | 2014-01-04 18:10:54
2014-01-04 18:10:54.489 | 2014-01-04 18:10:54 [pbr] In git context, generating filelist from git
2014-01-04 18:10:54.493 | 2014-01-04 18:10:54 warning: no previously-included files matching '*.pyc' found anywhere in distribution
2014-01-04 18:10:54.497 | 2014-01-04 18:10:54 Error in atexit._run_exitfuncs:
2014-01-04 18:10:54.501 | 2014-01-04 18:10:54 Traceback (most recent call last):
2014-01-04 18:10:54.505 | 2014-01-04 18:10:54 File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
2014-01-04 18:10:54.509 | 2014-01-04 18:10:54 func(*targs, **kargs)
2014-01-04 18:10:57.526 | 2014-01-04 18:10:57 File "/usr/lib/python2.7/multiprocessing/util.py", line 284, in _exit_function
2014-01-04 18:10:57.530 | 2014-01-04 18:10:57 info('process shutting down')
2014-01-04 18:10:57.533 | 2014-01-04 18:10:57 TypeError: 'NoneType' object is not callable
2014-01-04 18:10:57.537 | 2014-01-04 18:10:57 Error in sys.exitfunc:
2014-01-04 18:10:57.541 | 2014-01-04 18:10:57 Traceback (most recent call last):
2014-01-04 18:10:57.562 | 2014-01-04 18:10:57 File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
2014-01-04 18:10:57.562 | 2014-01-04 18:10:57 func(*targs, **kargs)
2014-01-04 18:10:57.563 | 2014-01-04 18:10:57 File "/usr/lib/python2.7/multiprocessing/util.py", line 284, in _exit_function
2014-01-04 18:10:57.563 | 2014-01-04 18:10:57 info('process shutting down')
2014-01-04 18:10:57.564 | 2014-01-04 18:10:57 TypeError: 'NoneType' object is not callable

I also see the following error:
2014-01-04 18:13:47.079 | ERROR: There was a conflict when trying to complete your request. (HTTP 409) (Request-ID: req-16cd29b2-19d8-4ee2-bb7b-5b39345a4fae)
According to the last update of https://bugs.launchpad.net/grenade/+bug/1245431, this shouldn't affect ci any more but I still got this error.

Also the following error:
2014-01-04 18:14:06.382 | + nova delete ex-float
2014-01-04 18:14:07.040 | ERROR: Unable to delete any of the specified servers.
2014-01-04 18:14:07.040 | No server with a name or ID of 'ex-float' exists.
2014-01-04 18:14:07.061 | + true
2014-01-04 18:14:07.061 | + timeout 90 sh -c 'while nova show ex-float; do sleep 1; done'
2014-01-04 18:14:07.825 | ERROR: No server with a name or ID of 'ex-float' exists.

[Xing Yang (xing-yang)]

I don't know if the first error regarding python-keystoneclient installation is a real problem. The real failure appeared to be the 2nd error related to aggregates:

2014-01-04 18:13:47.079 | ERROR: There was a conflict when trying to complete your request. (HTTP 409) (Request-ID: req-16cd29b2-19d8-4ee2-bb7b-5b39345a4fae)

[Jeremy Stanley (fungi)]

Some sort of bitrot has caused DevStack aggregates exercises to start failing for stable/grizzly (which in turn affects Grenade jobs for stable/havana changes).

[Xing Yang (xing-yang)]

This bug is affecting the following reviews and more:

https://review.openstack.org/#/c/64950/
https://review.openstack.org/#/c/64958/
https://review.openstack.org/#/c/64962/
https://review.openstack.org/#/c/64848/
https://review.openstack.org/#/c/64949/
https://review.openstack.org/#/c/64949/
https://review.openstack.org/#/c/65037/
https://review.openstack.org/#/c/64949/
https://review.openstack.org/#/c/65020/
........

Is there another bug number that this one should link to?

[Matt Riedemann (mriedem)]

From reading through the two bugs and comparing the logs, I'm pretty sure this is a duplicate of bug 1245431.

[Xing Yang (xing-yang)]

You are right. The last update (12/10/13) of bug 1245431 said "no longer affects: openstack-ci". Looks like the problem came back.

[Jeremy Stanley (fungi)]

No, the "no longer affects: openstack-ci" was because it's not a bug in openstack-ci. It's probably a bug on the stable/grizzly branch of whatever project(s) DevStack is using in the failing exercises.

[Jeremy Stanley (fungi)]

It's also possible that a client library involved in the failing exercise has changed itself in a way which is incompatible with the stable/grizzly branch of some server project, or possibly one of the declared requirements has recently released a new version which is only causing problems for grizzly deployments.

[Tiantian Gao (gtt116)]

It seems like cinder-scheduler was broken by netaddr conflict.

http://logs.openstack.org/24/61924/1/gate/gate-grenade-dsvm/0f8f694/logs/old/screen-c-sch.txt.gz

2014-01-08 18:10:30 ERROR [stevedore.extension] (netaddr 0.7.5 (/usr/lib/python2.7/dist-packages), Requirement.parse('netaddr>=0.7.6'))
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 134, in _load_plugins
    invoke_kwds,
  File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 146, in _load_one_plugin
    plugin = ep.load()
  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 2028, in load
    if require: self.require(env, installer)
  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 2041, in require
    working_set.resolve(self.dist.requires(self.extras),env,installer)))
  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 576, in resolve
    raise VersionConflict(dist,req) # XXX put more info here

[Tiantian Gao (gtt116)]

IMHO because python-netaddr deb package was installed first which version is 0.7.5. Then using old pip can not upgrade netaddr to higher version to match requirement.

I'm afraid we should back-port https://review.openstack.org/#/c/39827/ to stable/grizzly, even to stable/havana.

[Morgan Fainberg (mdrnstm)]

This seems to be blocking patches for stable/havana. This includes https://review.openstack.org/#/c/66149/ which is a high priority fix to make stable/havana and memcache tokens better.

[Alan Pevec (apevec)]

> I'm afraid we should back-port https://review.openstack.org/#/c/39827/

Anyone working on this backport?

[Alan Pevec (apevec)]

Should this be closed as a duplicate of bug 1267950 ?

[Alan Pevec (apevec)]

> Should this be closed as a duplicate of bug 1267950 ?

I mean, vice versa - done.

[Chmouel Boudjnah (chmouel)]

I have added a backport for grizzly here: https://review.openstack.org/#/c/39827/

[Chmouel Boudjnah (chmouel)]

or here i mean:

https://review.openstack.org/#/c/67311/

[Xavier Queralt (xqueralt-deactivatedaccount)]

This looks identical to bug 1207162

I've just backported its fix to grizzly and none of the jenkins jobs failed. What I'm still wondering is why this problem appeared right now. The requirements in grizzly haven't been updated for some time, could it be an update in ubuntu precise?

See https://review.openstack.org/#/c/67425 for the backport.

[Xing Yang (xing-yang)]

Have you backported the fix to Havana?

[Xavier Queralt (xqueralt-deactivatedaccount)]

The fix is already in Havana. The problem with the Havana reviews is the grenade test which, in order to test the update process, installs grizzly first and fails without this patch.

[Xavier Queralt (xqueralt-deactivatedaccount)]

I was obviously referring to the VersionConflict problem which makes cinder-scheduler unusable.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/67425
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=ee29648991aaaf80d4219e4ee6ea93edf9477f51
Submitter: Jenkins
Branch: stable/grizzly

commit ee29648991aaaf80d4219e4ee6ea93edf9477f51
Author: Monty Taylor <email address hidden>
Date: Fri Aug 2 15:43:47 2013 -0400

    Install things in setup_develop with pip -e

    We have some complex logic in here to try to do the right things with
    the requirements before doing the install of the package which still
    winds up being wrong in some cases. Since having written this code,
    we've learned that the logic we're trying to achieve is actually what
    pip install -e does. So just use that. We have to follow up with a chown
    of the resulting egg-info directory, because the sudo command will cause
    it to be written by root, which prevents subsequent commands from
    operating without privilege in the directory.

    (cherry picked from commit 408a4a7d1c24322b35f9a8617c7c62adeeee0dbe)

    Conflicts:
     functions

    Closes-Bug: #1266094
    Change-Id: Iffd068c94ef84475ebb30758bcf612075d225bea

[Alan Pevec (apevec)]

https://review.openstack.org/67739 fixes aggregates exercise in devstack grizzly