check_grenade_dsvm failed - aggregates tests failed

Bug #1266094 reported by Xing Yang on 2014-01-04
40
This bug affects 6 people
Affects Status Importance Assigned to Milestone
devstack
Undecided
Unassigned

Bug Description

Multiple errors in check_grenade_dsvm logs:
http://logs.openstack.org/58/64958/3/check/check-grenade-dsvm/5133512/console.html

The real failure seems to be related to "aggregates":
2014-01-04 18:15:59.936 | FAILED aggregates

I see the following errors:
2014-01-04 18:10:54.459 | 2014-01-04 18:10:54 Downloading/unpacking python-keystoneclient>=0.2.1,<0.3 (from -r keystone.egg-info/requires.txt (line 13))
2014-01-04 18:10:54.462 | 2014-01-04 18:10:54 Storing download in cache at /var/cache/pip/http%3A%2F%2Fpypi.openstack.org%2Fopenstack%2Fpython-keystoneclient%2Fpython-keystoneclient-0.2.5.tar.gz
2014-01-04 18:10:54.466 | 2014-01-04 18:10:54 Running setup.py egg_info for package python-keystoneclient
2014-01-04 18:10:54.470 | 2014-01-04 18:10:54
2014-01-04 18:10:54.474 | 2014-01-04 18:10:54 Installed /opt/stack/old/keystone/build/python-keystoneclient/d2to1-0.2.11-py2.7.egg
2014-01-04 18:10:54.478 | 2014-01-04 18:10:54 [pbr] Processing SOURCES.txt
2014-01-04 18:10:54.481 | 2014-01-04 18:10:54 warning: LocalManifestMaker: standard file '-c' not found
2014-01-04 18:10:54.485 | 2014-01-04 18:10:54
2014-01-04 18:10:54.489 | 2014-01-04 18:10:54 [pbr] In git context, generating filelist from git
2014-01-04 18:10:54.493 | 2014-01-04 18:10:54 warning: no previously-included files matching '*.pyc' found anywhere in distribution
2014-01-04 18:10:54.497 | 2014-01-04 18:10:54 Error in atexit._run_exitfuncs:
2014-01-04 18:10:54.501 | 2014-01-04 18:10:54 Traceback (most recent call last):
2014-01-04 18:10:54.505 | 2014-01-04 18:10:54 File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
2014-01-04 18:10:54.509 | 2014-01-04 18:10:54 func(*targs, **kargs)
2014-01-04 18:10:57.526 | 2014-01-04 18:10:57 File "/usr/lib/python2.7/multiprocessing/util.py", line 284, in _exit_function
2014-01-04 18:10:57.530 | 2014-01-04 18:10:57 info('process shutting down')
2014-01-04 18:10:57.533 | 2014-01-04 18:10:57 TypeError: 'NoneType' object is not callable
2014-01-04 18:10:57.537 | 2014-01-04 18:10:57 Error in sys.exitfunc:
2014-01-04 18:10:57.541 | 2014-01-04 18:10:57 Traceback (most recent call last):
2014-01-04 18:10:57.562 | 2014-01-04 18:10:57 File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
2014-01-04 18:10:57.562 | 2014-01-04 18:10:57 func(*targs, **kargs)
2014-01-04 18:10:57.563 | 2014-01-04 18:10:57 File "/usr/lib/python2.7/multiprocessing/util.py", line 284, in _exit_function
2014-01-04 18:10:57.563 | 2014-01-04 18:10:57 info('process shutting down')
2014-01-04 18:10:57.564 | 2014-01-04 18:10:57 TypeError: 'NoneType' object is not callable

I also see the following error:
2014-01-04 18:13:47.079 | ERROR: There was a conflict when trying to complete your request. (HTTP 409) (Request-ID: req-16cd29b2-19d8-4ee2-bb7b-5b39345a4fae)
According to the last update of https://bugs.launchpad.net/grenade/+bug/1245431, this shouldn't affect ci any more but I still got this error.

Also the following error:
2014-01-04 18:14:06.382 | + nova delete ex-float
2014-01-04 18:14:07.040 | ERROR: Unable to delete any of the specified servers.
2014-01-04 18:14:07.040 | No server with a name or ID of 'ex-float' exists.
2014-01-04 18:14:07.061 | + true
2014-01-04 18:14:07.061 | + timeout 90 sh -c 'while nova show ex-float; do sleep 1; done'
2014-01-04 18:14:07.825 | ERROR: No server with a name or ID of 'ex-float' exists.

Xing Yang (xing-yang) wrote :

I don't know if the first error regarding python-keystoneclient installation is a real problem. The real failure appeared to be the 2nd error related to aggregates:

2014-01-04 18:13:47.079 | ERROR: There was a conflict when trying to complete your request. (HTTP 409) (Request-ID: req-16cd29b2-19d8-4ee2-bb7b-5b39345a4fae)

Jeremy Stanley (fungi) wrote :

Some sort of bitrot has caused DevStack aggregates exercises to start failing for stable/grizzly (which in turn affects Grenade jobs for stable/havana changes).

affects: openstack-ci → devstack
Matt Riedemann (mriedem) wrote :

From reading through the two bugs and comparing the logs, I'm pretty sure this is a duplicate of bug 1245431.

Xing Yang (xing-yang) wrote :

You are right. The last update (12/10/13) of bug 1245431 said "no longer affects: openstack-ci". Looks like the problem came back.

Jeremy Stanley (fungi) wrote :

No, the "no longer affects: openstack-ci" was because it's not a bug in openstack-ci. It's probably a bug on the stable/grizzly branch of whatever project(s) DevStack is using in the failing exercises.

Jeremy Stanley (fungi) wrote :

It's also possible that a client library involved in the failing exercise has changed itself in a way which is incompatible with the stable/grizzly branch of some server project, or possibly one of the declared requirements has recently released a new version which is only causing problems for grizzly deployments.

Tiantian Gao (gtt116) wrote :

It seems like cinder-scheduler was broken by netaddr conflict.

http://logs.openstack.org/24/61924/1/gate/gate-grenade-dsvm/0f8f694/logs/old/screen-c-sch.txt.gz

2014-01-08 18:10:30 ERROR [stevedore.extension] (netaddr 0.7.5 (/usr/lib/python2.7/dist-packages), Requirement.parse('netaddr>=0.7.6'))
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 134, in _load_plugins
    invoke_kwds,
  File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 146, in _load_one_plugin
    plugin = ep.load()
  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 2028, in load
    if require: self.require(env, installer)
  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 2041, in require
    working_set.resolve(self.dist.requires(self.extras),env,installer)))
  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 576, in resolve
    raise VersionConflict(dist,req) # XXX put more info here

Tiantian Gao (gtt116) wrote :

IMHO because python-netaddr deb package was installed first which version is 0.7.5. Then using old pip can not upgrade netaddr to higher version to match requirement.

I'm afraid we should back-port https://review.openstack.org/#/c/39827/ to stable/grizzly, even to stable/havana.

Morgan Fainberg (mdrnstm) wrote :

This seems to be blocking patches for stable/havana. This includes https://review.openstack.org/#/c/66149/ which is a high priority fix to make stable/havana and memcache tokens better.

Alan Pevec (apevec) wrote :

> I'm afraid we should back-port https://review.openstack.org/#/c/39827/

Anyone working on this backport?

Alan Pevec (apevec) wrote :

Should this be closed as a duplicate of bug 1267950 ?

tags: added: grizzly-backport-potential havana-backport-potential
Alan Pevec (apevec) wrote :

> Should this be closed as a duplicate of bug 1267950 ?

I mean, vice versa - done.

Tiantian Gao (gtt116) on 2014-01-16
Changed in devstack:
status: New → Confirmed
Chmouel Boudjnah (chmouel) wrote :

I have added a backport for grizzly here: https://review.openstack.org/#/c/39827/

This looks identical to bug 1207162

I've just backported its fix to grizzly and none of the jenkins jobs failed. What I'm still wondering is why this problem appeared right now. The requirements in grizzly haven't been updated for some time, could it be an update in ubuntu precise?

See https://review.openstack.org/#/c/67425 for the backport.

Xing Yang (xing-yang) wrote :

Have you backported the fix to Havana?

The fix is already in Havana. The problem with the Havana reviews is the grenade test which, in order to test the update process, installs grizzly first and fails without this patch.

I was obviously referring to the VersionConflict problem which makes cinder-scheduler unusable.

Reviewed: https://review.openstack.org/67425
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=ee29648991aaaf80d4219e4ee6ea93edf9477f51
Submitter: Jenkins
Branch: stable/grizzly

commit ee29648991aaaf80d4219e4ee6ea93edf9477f51
Author: Monty Taylor <email address hidden>
Date: Fri Aug 2 15:43:47 2013 -0400

    Install things in setup_develop with pip -e

    We have some complex logic in here to try to do the right things with
    the requirements before doing the install of the package which still
    winds up being wrong in some cases. Since having written this code,
    we've learned that the logic we're trying to achieve is actually what
    pip install -e does. So just use that. We have to follow up with a chown
    of the resulting egg-info directory, because the sudo command will cause
    it to be written by root, which prevents subsequent commands from
    operating without privilege in the directory.

    (cherry picked from commit 408a4a7d1c24322b35f9a8617c7c62adeeee0dbe)

    Conflicts:
     functions

    Closes-Bug: #1266094
    Change-Id: Iffd068c94ef84475ebb30758bcf612075d225bea

tags: added: in-stable-grizzly
Alan Pevec (apevec) wrote :

https://review.openstack.org/67739 fixes aggregates exercise in devstack grizzly

Alan Pevec (apevec) on 2014-01-24
Changed in devstack:
status: Confirmed → Fix Committed
Dean Troyer (dtroyer) on 2014-01-27
Changed in devstack:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers