Stack trace when stack.sh hits "cinder-manage db sync"

[Ash Wilson]

I'm attempting to install Icehouse on an Ubuntu 14.04 LTS (Trusty Tahr) server, but stack.sh is dying mid-install with the following stack:

2014-05-05 20:57:12.739 | + /usr/local/bin/cinder-manage db sync
2014-05-05 20:57:12.740 | Traceback (most recent call last):
2014-05-05 20:57:12.742 | File "/usr/local/bin/cinder-manage", line 10, in <module>
2014-05-05 20:57:12.744 | execfile(__file__)
2014-05-05 20:57:12.746 | File "/opt/stack/cinder/bin/cinder-manage", line 545, in <module>
2014-05-05 20:57:12.748 | main()
2014-05-05 20:57:12.749 | File "/opt/stack/cinder/bin/cinder-manage", line 525, in main
2014-05-05 20:57:12.751 | version=version.version_string())
2014-05-05 20:57:12.753 | File "/usr/local/lib/python2.7/dist-packages/oslo/config/cfg.py", line 1605, in __call__
2014-05-05 20:57:12.755 | else sys.argv[1:])
2014-05-05 20:57:12.756 | File "/usr/local/lib/python2.7/dist-packages/oslo/config/cfg.py", line 2144, in _parse_cli_opts
2014-05-05 20:57:12.758 | opt._add_to_cli(self._oparser, group)
2014-05-05 20:57:12.760 | File "/usr/local/lib/python2.7/dist-packages/oslo/config/cfg.py", line 986, in _add_to_cli
2014-05-05 20:57:12.761 | self.handler(subparsers)
2014-05-05 20:57:12.763 | File "/opt/stack/cinder/bin/cinder-manage", line 461, in add_command_parsers
2014-05-05 20:57:12.764 | command_object = CATEGORIES[category]()
2014-05-05 20:57:12.766 | File "/opt/stack/cinder/bin/cinder-manage", line 250, in __init__
2014-05-05 20:57:12.767 | rpc.init(CONF)
2014-05-05 20:57:12.769 | File "/opt/stack/cinder/cinder/rpc.py", line 63, in init
2014-05-05 20:57:12.770 | aliases=TRANSPORT_ALIASES)
2014-05-05 20:57:12.772 | File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 183, in get_transport
2014-05-05 20:57:12.774 | invoke_kwds=kwargs)
2014-05-05 20:57:12.775 | File "/usr/local/lib/python2.7/dist-packages/stevedore/driver.py", line 45, in __init__
2014-05-05 20:57:12.777 | verify_requirements=verify_requirements,
2014-05-05 20:57:12.778 | File "/usr/local/lib/python2.7/dist-packages/stevedore/named.py", line 55, in __init__
2014-05-05 20:57:12.780 | verify_requirements)
2014-05-05 20:57:12.782 | File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 170, in _load_plugins
2014-05-05 20:57:12.783 | self._on_load_failure_callback(self, ep, err)
2014-05-05 20:57:12.785 | File "/usr/local/lib/python2.7/dist-packages/stevedore/driver.py", line 50, in _default_on_load_failure
2014-05-05 20:57:12.786 | raise err
2014-05-05 20:57:12.788 | AttributeError: 'Module_six_moves_urllib_parse' object has no attribute 'SplitResult'

[Ash Wilson (ash-wilson)]

Interesting. I was able to reproduce this from the command line with:

cinder-manage help

But I can fix it by hand by running:

sudo apt-get install libxslt1-dev
sudo pip install --upgrade -r /opt/stack/cinder/requirements.txt

Is the issue just that libxslt1-dev is missing?

[Ash Wilson (ash-wilson)]

Here's my local.conf as well:

[[local|localrc]]
ADMIN_PASSWORD=redacted
MYSQL_PASSWORD=redacted
RABBIT_PASSWORD=redacted
SERVICE_PASSWORD=redacted
SERVICE_TOKEN=redactedredacted

SWIFT_HASH=000000000000

# Services
ENABLED_SERVICES=rabbit,mysql,key
ENABLED_SERVICES+=,n-api,n-crt,n-obj,n-cpu,n-cond,n-sch,n-novnc,n-cauth
ENABLED_SERVICES+=,neutron,q-svc,q-agt,q-dhcp,q-l3,q-meta,q-lbaas
ENABLED_SERVICES+=,s-proxy,s-object,s-container,s-account
ENABLED_SERVICES+=,g-api,g-reg
ENABLED_SERVICES+=,cinder,c-api,c-vol,c-sch,c-bak
ENABLED_SERVICES+=,heat,h-api,h-api-cfn,h-api-cw,h-eng
ENABLED_SERVICES+=,trove,tr-api,tr-tmgr,tr-cond
ENABLED_SERVICES+=,horizon

# Images
# Use this image when creating test instances
IMAGE_URLS+=",http://cdn.download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-disk.img"
# Use this image when working with Orchestration (Heat)
IMAGE_URLS+=",http://fedorapeople.org/groups/heat/prebuilt-jeos-images/F17-x86_64-cfntools.qcow2"

# Branches
NOVA_BRANCH=stable/icehouse
CINDER_BRANCH=stable/icehouse
GLANCE_BRANCH=stable/icehouse
HORIZON_BRANCH=stable/icehouse
KEYSTONE_BRANCH=stable/icehouse
NEUTRON_BRANCH=stable/icehouse
SWIFT_BRANCH=stable/icehouse

# Enable Logging
LOGFILE=/opt/stack/logs/stack.sh.log
VERBOSE=True
LOG_COLOR=True
SCREEN_LOGDIR=/opt/stack/logs

[Attila Fazekas (afazekas)]

I can see the same issue on f20 with six 1.5.2.

cinder/openstack/common/network_utils.py has a reference to SplitResult.

# pip install -U six
Upgrades six to 1.6.1 and it works.

According to the https://github.com/openstack/requirements/blob/stable/icehouse/global-requirements.txt 1.5.2 should be fine, but it is not :(

[Matt Riedemann (mriedem)]

What is the database url in your cinder.conf file? I'm running stable/icehouse with RHEL 6.5 and python-six 1.5.2 and don't have this issue running cinder-manage db sync. I'm not using devstack though.

[Jay Bryant (jsbryant)]

I just tried this on an ubuntu 12.10 system, using devstack and didn't see this issue. So, I would have concerns about trying to merge this into Cinder without more information.

[Matt Riedemann (mriedem)]

What version of stevedore is installed when this blows up?

[Matt Riedemann (mriedem)]

I was wondering about stevedore because of this:

http://lists.openstack.org/pipermail/openstack-dev/2014-January/025602.html

And it's stevedore>=0.14 in cinder for stable/icehouse:

http://lists.openstack.org/pipermail/openstack-dev/2014-January/025602.html

I'm running with stevedore 0.14.1 and not seeing this problem on stable/icehouse.

[Ash Wilson (ash-wilson)]

I was able to work around this by manually updating six, as afazekas suggested, and re-running stack.sh.

It's probably worth noting that I was getting the same stack trace running any cinder-manage command, even "help".

Stevedore version:

stevedore (0.15.3.g3668de2)

Oddly, I don't appear to have a cinder.conf. Er, where would I find it? I checked under /opt/stack/cinder with find.

[Doug Hellmann (doug-hellmann)]

Stevedore is throwing the error from its driver loading code. It isn't trying to access the missing attribute itself. Unfortunately the way the error handling is implemented the true source of the exception is being masked.

[Attila Fazekas (afazekas)]

The https://github.com/openstack/stevedore does not have stable/icehouse branch, so I am using the master.

[Derek Anderson (derek-anderson)]

I did a bit of bisecting, it looks like this was introduced in:

https://github.com/openstack/oslo.messaging/commit/13608437bb8d491badbc6c906f8d60b51e52cd79

but that just involved adding six to replace oslo.messaging's url parser.

The django repo shows that 1.6 six.urllib.parse is missing that SplitResult attribute:

https://github.com/django/django/blob/1.6/django/utils/six.py

[Derek Anderson (derek-anderson)]

Correction, six.py got support for SplitResult in the 1.6 series.

1.5.2 and earlier are missing it:
https://bitbucket.org/gutworth/six/src/9ca411aad9c04394e441ccc01841bf53320ecac0/six.py?at=1.5.2

[Hua Zhang (zhhuabj)]

I can see the same issue on ubuntu 14.04 with below modules:
$ sudo pip list |grep stevedore
stevedore (0.15.7.gff1f0fd)
$ sudo pip list |grep ^six
six (1.6.1)

[netra (netra-freesystems)]

After running Server for 3 days successfully getting above error . No nova and cinder services are starting .

Any fix for it??

[Rashid Rana (ranamin)]

the correct package that contains SplitResult is 1.6.1. Please look for conflicting versions of this package. I had 1.5.2 under /usr/lib/python2.7/dist-packages which was taking precedence over /usr/local/lib/python2.7/dist-packages. You can fix it either by pointing PYTHONPATH to correct version or just overwrite six.py with 1.6.1 version from /usr/local/lib/python2.7/dist-packages.

[Rashid Rana (ranamin)]

How do you plan to fix it?

[Matt Riedemann (mriedem)]

I think I figured this out.

We're not hitting this with six 1.5.2 because we're using oslo.messaging 1.3.0a9 which didn't have this change:

https://github.com/openstack/oslo.messaging/commit/13608437bb8d491badbc6c906f8d60b51e52cd79

I'm not sure what was being used in the bug reported, i.e. is RDO building from trunk for it's client packages in Icehouse?

Anyway, that's why we didn't have this issue.

Also, the requirement for oslo.messaging in stable/icehouse is this:

oslo.messaging>=1.3.0a9

And 13608437bb8d491badbc6c906f8d60b51e52cd79 is not in the 1.3.0 release of oslo.messaging, which is the latest release, so someone must be building from trunk of oslo.messaging - and this means that oslo.messaging should have it's requirements.txt file updated for six since it's using that feature that's only in >= 1.6.0.

[Brant Knudson (blk-u)]

oslo.messaging requirements.txt has six>=1.6.0 -- http://git.openstack.org/cgit/openstack/oslo.messaging/tree/requirements.txt#n8

[Hua Zhang (zhhuabj)]

the problem disappear after git pull the latest oslo.messaging requirements.txt as Matt and Brant said above

[Hua Zhang (zhhuabj)]

my above problem (comment #20) is caused by great wall firewall of China government, all problem disappear after I use vpn, thanks.

[Aaron Rosen (arosen)]

rm -fr /usr/lib/python2.7/dist-packages/six.py* Fixed it for me as Rashid Rana in #16 pointed out.

[Doug Hellmann (doug-hellmann)]

This looks like it was a configuration issue, rather than a bug. If I'm wrong, please reopen the ticket.