check_grenade_dsvm failed aggregate devstack exercise

[Chen Xiao]

Log: http://logs.openstack.org/45/74245/1/check/check-grenade-dsvm/cd7486c/console.html
In stable/havana branch,
2014-02-18 03:47:52.829 | FAILED client-args
2014-02-18 03:47:52.829 | FAILED swift
2014-02-18 03:47:52.829 | =====================================================================

another error,

2014-02-18 03:46:03.140 | + nova aggregate-add-host 2 devstack-precise-1391998468
2014-02-18 03:46:03.737 | ERROR: There was a conflict when trying to complete your request. (HTTP 409) (Request-ID: req-df691d40-93f0-4b15-8303-db7ceb4884c1)

2014-02-18 03:43:08.564 | 2014-02-18 03:43:08 Downloading/unpacking python-keystoneclient>=0.2.1,<0.3 (from keystone==2013.1.5.a4.g8fcc18c)
2014-02-18 03:43:08.564 | 2014-02-18 03:43:08 Running setup.py egg_info for package python-keystoneclient
2014-02-18 03:43:08.564 | 2014-02-18 03:43:08
2014-02-18 03:43:08.565 | 2014-02-18 03:43:08 Installed /tmp/pip-build-root/python-keystoneclient/pbr-0.5.23-py2.7.egg
2014-02-18 03:43:08.565 | 2014-02-18 03:43:08 Searching for d2to1>=0.2.10,<0.3
2014-02-18 03:43:08.565 | 2014-02-18 03:43:08 Reading http://pypi.openstack.org/openstack/d2to1/
2014-02-18 03:43:08.565 | 2014-02-18 03:43:08 Reading http://pypi.openstack.org/openstack/d2to1/?C=M;O=A
2014-02-18 03:43:08.566 | 2014-02-18 03:43:08 Reading http://pypi.openstack.org/openstack/d2to1/?C=N;O=D
2014-02-18 03:43:08.566 | 2014-02-18 03:43:08 Reading http://pypi.openstack.org/openstack/d2to1/?C=D;O=A
2014-02-18 03:43:08.566 | 2014-02-18 03:43:08 Reading http://pypi.openstack.org/openstack/d2to1/?C=S;O=A
2014-02-18 03:43:08.566 | 2014-02-18 03:43:08 Best match: d2to1 0.2.11
2014-02-18 03:43:08.567 | 2014-02-18 03:43:08 Downloading http://pypi.openstack.org/openstack/d2to1/d2to1-0.2.11.tar.gz
2014-02-18 03:43:08.567 | 2014-02-18 03:43:08 Processing d2to1-0.2.11.tar.gz
2014-02-18 03:43:08.567 | 2014-02-18 03:43:08 Writing /tmp/easy_install-ohmSu6/d2to1-0.2.11/setup.cfg
2014-02-18 03:43:08.567 | 2014-02-18 03:43:08 Running d2to1-0.2.11/setup.py -q bdist_egg --dist-dir /tmp/easy_install-ohmSu6/d2to1-0.2.11/egg-dist-tmp-LUmScr
2014-02-18 03:43:08.567 | 2014-02-18 03:43:08
2014-02-18 03:43:08.568 | 2014-02-18 03:43:08 Installed /tmp/pip-build-root/python-keystoneclient/d2to1-0.2.11-py2.7.egg
2014-02-18 03:43:08.569 | 2014-02-18 03:43:08 [pbr] Excluding argparse: Python 2.6 only dependency
2014-02-18 03:43:08.572 | 2014-02-18 03:43:08 [pbr] Processing SOURCES.txt
2014-02-18 03:43:08.574 | 2014-02-18 03:43:08 warning: LocalManifestMaker: standard file '-c' not found
2014-02-18 03:43:08.577 | 2014-02-18 03:43:08
2014-02-18 03:43:08.579 | 2014-02-18 03:43:08 warning: no previously-included files found matching '.gitignore'
2014-02-18 03:43:08.582 | 2014-02-18 03:43:08 warning: no previously-included files found matching '.gitreview'
2014-02-18 03:43:08.585 | 2014-02-18 03:43:08 warning: no previously-included files matching '*.pyc' found anywhere in distribution
2014-02-18 03:43:08.587 | 2014-02-18 03:43:08 Error in atexit._run_exitfuncs:
2014-02-18 03:43:08.589 | 2014-02-18 03:43:08 Traceback (most recent call last):
2014-02-18 03:43:08.591 | 2014-02-18 03:43:08 File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
2014-02-18 03:43:08.594 | 2014-02-18 03:43:08 func(*targs, **kargs)
2014-02-18 03:43:08.602 | 2014-02-18 03:43:08 File "/usr/lib/python2.7/multiprocessing/util.py", line 284, in _exit_function
2014-02-18 03:43:08.602 | 2014-02-18 03:43:08 info('process shutting down')
2014-02-18 03:43:08.602 | 2014-02-18 03:43:08 TypeError: 'NoneType' object is not callable
2014-02-18 03:43:08.603 | 2014-02-18 03:43:08 Error in atexit._run_exitfuncs:
2014-02-18 03:43:08.606 | 2014-02-18 03:43:08 Traceback (most recent call last):
2014-02-18 03:43:08.608 | 2014-02-18 03:43:08 File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
2014-02-18 03:43:08.610 | 2014-02-18 03:43:08 func(*targs, **kargs)
2014-02-18 03:43:08.613 | 2014-02-18 03:43:08 File "/usr/lib/python2.7/multiprocessing/util.py", line 284, in _exit_function
2014-02-18 03:43:08.615 | 2014-02-18 03:43:08 info('process shutting down')
2014-02-18 03:43:08.618 | 2014-02-18 03:43:08 TypeError: 'NoneType' object is not callable
2014-02-18 03:43:08.620 | 2014-02-18 03:43:08 Error in sys.exitfunc:
2014-02-18 03:43:08.623 | 2014-02-18 03:43:08 Traceback (most recent call last):
2014-02-18 03:43:08.625 | 2014-02-18 03:43:08 File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
2014-02-18 03:43:08.628 | 2014-02-18 03:43:08 func(*targs, **kargs)
2014-02-18 03:43:12.359 | 2014-02-18 03:43:12 File "/usr/lib/python2.7/multiprocessing/util.py", line 284, in _exit_function
2014-02-18 03:43:12.361 | 2014-02-18 03:43:12 info('process shutting down')
2014-02-18 03:43:12.397 | 2014-02-18 03:43:12 TypeError: 'NoneType' object is not callable

[Stuart McLaren (stuart-mclaren)]

I may have hit this too

http://logs.openstack.org/45/74245/1/check/check-grenade-dsvm/cd7486c/console.html

2014-02-18 03:46:09.620 | Test Swift
2014-02-18 03:46:09.620 | + swift --os-tenant-name=demo --os-username=demo --os-password=secret --os-auth-url=http://127.0.0.1:5000/v2.0 stat
2014-02-18 03:46:09.804 | Traceback (most recent call last):
2014-02-18 03:46:09.842 | File "/usr/local/bin/swift", line 35, in <module>
2014-02-18 03:46:09.843 | from swiftclient import Connection, HTTPException
2014-02-18 03:46:09.843 | ImportError: cannot import name HTTPException

[Matt Riedemann (mriedem)]

See this in the logs:

http://logs.openstack.org/45/74245/1/check/check-grenade-dsvm/7d75b3a/logs/old/error.txt.gz

[ERROR] /opt/stack/old/devstack/exercises/swift.sh:48 Failure geting status

Seeing this in the screen-s-proxy logs:

TypeError: int() argument must be a string or a number, not 'NoneType' (txn: tx4ba9eeaf4a694e8880563c8b1fbc2f68)

Unfortunately it looks like we can't fingerprint this in logstash because logs/screen-s-proxy.txt.gz isn't indexed, due to it's formatting (lack of timestamps is the problem I think):

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiYXJndW1lbnQgbXVzdCBiZSBhIHN0cmluZyBvciBhIG51bWJlciwgbm90XCIgQU5EIGZpbGVuYW1lOmxvZ3Mqc2NyZWVuLXMtcHJveHkudHh0Lmd6IiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzkyOTE0NjAxMjYxfQ==

[Matt Riedemann (mriedem)]

Sounds like we need this: https://review.openstack.org/#/c/74451/

[Matt Riedemann (mriedem)]

Looks like we could query on syslog since swift sends stuff there, but then we don't have a 100% failure rate:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRXJyb3JcXDogaW50XFwoXFwpIGFyZ3VtZW50IG11c3QgYmUgYSBzdHJpbmcgb3IgYSBudW1iZXIsIG5vdCBcXCdOb25lVHlwZVxcJ1wiIEFORCBmaWxlbmFtZTpcImxvZ3Mvc3lzbG9nLnR4dFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzkyOTMyMzM0NDU0fQ==

[Alan Pevec (apevec)]

This bug includes multiple issues in description, we could we track only aggregate part here and use https://bugs.launchpad.net/openstack-ci/+bug/1281540 for swiftclient parts.

[Alan Pevec (apevec)]

Aggregate failure in http://logs.openstack.org/62/73562/1/check/check-grenade-dsvm/1080e25/console.html#_2014-02-27_08_52_41_637

+ nova aggregate-update 1 test_aggregate_19630
...
error: unrecognized arguments: test_aggregate_19630

[Alan Pevec (apevec)]

Broken by https://review.openstack.org/73526

[Alan Pevec (apevec)]

BTW this error from the original report:

2014-02-18 03:46:03.140 | + nova aggregate-add-host 2 devstack-precise-1391998468
2014-02-18 03:46:03.737 | ERROR: There was a conflict when trying to complete your request. (HTTP 409) (Request-ID: req-df691d40-93f0-4b15-8303-db7ceb4884c1)

is expected:
if nova aggregate-add-host $AGGREGATE2_ID $FIRST_HOST; then
    die $LINENO "could add duplicate host to single aggregate"
fi

[Alan Pevec (apevec)]

Options are:
- backport https://bugs.launchpad.net/grenade/+bug/1023131 to stable
(and remove devstack exercises from devstack)
- adjust devstack aggregates exercise on stable to work with novaclient master
- revert https://review.openstack.org/73526 - luckily it missed novaclient 2.16.0 release

[Alan Pevec (apevec)]

> - revert https://review.openstack.org/73526

Proposed in https://review.openstack.org/76841

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/76841
Committed: https://git.openstack.org/cgit/openstack/python-novaclient/commit/?id=360a3393f4d52746a9e87ac0011cc700a51965ad
Submitter: Jenkins
Branch: master

commit 360a3393f4d52746a9e87ac0011cc700a51965ad
Author: Alan Pevec <email address hidden>
Date: Thu Feb 27 13:12:41 2014 +0100

    Revert "'name' should as be optional param on aggregate-update"

    This reverts commit a2a1ef8f36644c3487bdaf00b2e70bcc6b948fb4.

    Change-Id: I43bdc254e391c4b20254b8732c772fc9e728310c
    Reopens-bug: #1280118
    Partial-bug: #1281416

[Thiago da Silva (thiagodasilva)]

Reading over the comments, it doesn't seem like this was a bug in swift or swiftclient code. The fixes proposed were to either devstack or the novaclient, plus the bug mentioned in comment #5 has the swiftclient bug marked as Invalid.

I propose we mark this bug as Invalid for Swift.