./stack.sh crashes on keystone requirements

[Tomasz Trębski]

After I updated my /devstack folder to the recent version from master I noticed that ./stack.sh fails when installing requirements specified under /opt/stack/keystone/requirements.txt.

Issue regards oslo.config library and occurs if entry is specified as follow:
oslo.config>=1.9.3,<1.10.0 # Apache-2.0
Unfortunately that fails with DistributionNotFound error and crashes script.
After I changed the line to:
oslo.config>=1.10.0
It seems that installing requirements for keystone passes.

I checked, prior to this change, that I have oslo.config already installed as 1.10.0.

Perhaps this is the wrong place to fill this bug as it may be keystone upstream issue, but I am a beginner in openstack and don't know about internal logic.

Summary:
Where: origin/master, HEAD SHA--> 1235581559087974580c0e5f6e3ade82e9030a70
What: keystone requirements failed to resolve due to oslo.config invalid entry
System: Linux fpi-virtual-013 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:43:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[Radoslaw Smigielski (radoslaw-smigielski)]

Tomku, do you see the same problem with the last devstack on fresh, clean system? I am guessing you may have some wired dependency problem caused by some (pre-)existing dependencies. I installed devstack from the latest code available today on clean system and all went fine.

[Tomasz Trębski (kornicameister)]

Hi Radek, as we speak I am launching clean snapshot where I have downloaded the latest code from origin/master. The system is raw and clean Ubuntu Server with only git installed as non-OOTB component.

Anyway if error will happen again I will try to add an attachment with ./stack.sh log to provide some better background to investigate the problem.

[Tomasz Trębski (kornicameister)]

At the bottom of the file there is a part where a keystone tries to resolve its dependencies and fail fast on oslo.config

[Tomasz Trębski (kornicameister)]

local.conf I am using right now

[Tomasz Trębski (kornicameister)]

local.conf I am using right now

[Tomasz Trębski (kornicameister)]

I managed to reproduce an error, I uploaded log and local.conf from the installation.
Sorry for duplicating comment with local.conf attachment

[Radek Pospisil (radek-pospisil)]

I've got the same issue - I cannot ./stack.sh devstack after release of oslo.config 1.10.0 .
I've tried to reclone, remove all pip packages before ./stack.sh ... what I've got is:

2015-04-16 11:58:46.024 | + [[ ! -d /opt/stack/status/stack ]]
2015-04-16 11:58:46.025 | ++ ls /opt/stack/status/stack/h-eng.failure
2015-04-16 11:58:46.027 | + failures=/opt/stack/status/stack/h-eng.failure
2015-04-16 11:58:46.027 | + for service in '$failures'
2015-04-16 11:58:46.027 | ++ basename /opt/stack/status/stack/h-eng.failure
2015-04-16 11:58:46.028 | + service=h-eng.failure
2015-04-16 11:58:46.029 | + service=h-eng
2015-04-16 11:58:46.029 | + echo 'Error: Service h-eng is not running'
2015-04-16 11:58:46.029 | Error: Service h-eng is not running
2015-04-16 11:58:46.029 | + '[' -n /opt/stack/status/stack/h-eng.failure ']'
2015-04-16 11:58:46.029 | + die 1381 'More details about the above errors can be found with screen, with ./rejoin-stack.sh'
2015-04-16 11:58:46.029 | + local exitcode=0
2015-04-16 11:58:46.029 | [Call Trace]
2015-04-16 11:58:46.029 | ./stack.sh:1342:service_check
2015-04-16 11:58:46.029 | /home/devstack/devstack/functions-common:1381:die
2015-04-16 11:58:46.033 | [ERROR] /home/devstack/devstack/functions-common:1381 More details about the above errors can be found with screen, with ./rejoin-stack.sh
2015-04-16 11:58:47.035 | Error on exit

It means that h-eng cannot be started:

devstack@mydpha061:~/devstack$ /opt/stack/heat/bin/heat-engine --config-file=/etc/heat/heat.conf & echo $! >/opt/stack/status/stack/h-eng.pid; fg || echo "h-eng failed to start" | tee "/opt/stack/status/stack/h-eng.failure"
[1] 7895
/opt/stack/heat/bin/heat-engine --config-file=/etc/heat/heat.conf
2015-04-16 13:52:00.673 7895 CRITICAL heat.engine [-] Could not load AWSTemplateFormatVersion.2010-09-09: (oslo.config 1.10.0 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('oslo.config<1.10.0,>=1.9.3'))
ERROR: No template format plugins registered
h-eng failed to start

[Radek Pospisil (radek-pospisil)]

IMO the issue is that during ./stack.sh each service is installed one-by-one, so it 'modifies' installed packages with respect to its requirements. I've observed, that during ./stack.sh the version of oslo.config was 1.10.0, then 1.9.3, after that 1.10.0 ...
So when the services are starting, the requirements are not fulfilled (see also my previous comment) - the service fails:

/opt/stack/heat/bin/heat-engine --config-file=/etc/heat/heat.conf
2015-04-16 13:52:00.673 7895 CRITICAL heat.engine [-] Could not load AWSTemplateFormatVersion.2010-09-09: (oslo.config 1.10.0 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('oslo.config<1.10.0,>=1.9.3'))

according to https://git.openstack.org/openstack/requirements.git all services has to have:
oslo.config>=1.9.3,<1.10.0 # Apache-2.0

In my devstack all services have same (global one) oslo.config requirement, except for mistral and os-collect-config

./horizon/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./murano/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./keystone/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./keystone/requirements-py3.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0

./os-collect-config/requirements.txt:oslo.config>=1.9.0 # Apache-2.0

./congress/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0

./mistral/requirements.txt:oslo.config>=1.4.0 # Apache-2.0

./glance/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./cinder/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./requirements/global-requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./requirements/tests/files/project-with-bad-requirement.txt:oslo.config>=1.1.0
./neutron/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./ceilometer/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./ceilometer/requirements-py3.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./heat/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./heat/heat_integrationtests/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./murano-dashboard/test-requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./nova/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0

./requirements/global-requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./neutron/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./neutron/neutron.egg-info/requires.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./ceilometer/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./ceilometer/ceilometer.egg-info/requires.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./ceilometer/requirements-py3.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./heat/requirements.txt:oslo.config>=1.9.3,<=1.10.0 # Apache-2.0
./heat/heat_integrationtests/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./heat/heat.egg-info/requires.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./murano-dashboard/test-requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./nova/requirements.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0
./nova/nova.egg-info/requires.txt:oslo.config>=1.9.3,<1.10.0 # Apache-2.0

[Robert Collins (lifeless)]

Pretty sure we fixed this by changing the use of load_entrypoints to disable requirement checking.