[detach-keystone] Deployment with detached-keystone failed - keystone is not available

[Vladimir Khlyunev]

ISO 506,
CI job https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.plugins.thread_2_separate_services/127/
Snapshot https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.plugins.thread_2_separate_services/127/artifact/logs/fail_error_separate_all_service-fuel-snapshot-2016-02-01_02-07-34.tar.xz

Steps:
1) Build&install detach-db/keystone/rabbitmq cluster
2) Deploy cluster: VLAN, 3 controllers, 3 database+keystone+rabbitmq, 1 compute+cinder

Result:
Multiple errors in puppet.log like:
Command: 'openstack ["endpoint", "create", "--format", "shell", ["keystone", "--region", "RegionOne", "--publicurl", "https://10.109.18.3:5000/v2.0", "--internalurl", "http://10.109.16.4:5000/v2.0", "--adminurl", "http://10.109.16.4:35357/v2.0"]]' has been running for more then 20 seconds!
------
Could not prefetch keystone_service provider 'openstack': Execution of '/usr/bin/openstack service list --quiet --format csv --long' returned 1:
-------
and so on

Looks like keystone is not available or blocked by firewall.

[Matthew Mosesohn (raytrac3r)]

This bug might be already fixed in https://review.openstack.org/274894 Waiting for tonight's SWARM to see if it's reproduced.

[Vladimir Kuklin (vkuklin)]

I will mark this bug as a duplicate of the similar bug - let's wait for swarm results verification

[Vladimir Khlyunev (vkhlyunev)]

The issue were reproduced - https://product-ci.infra.mirantis.net/view/8.0_swarm/job/8.0.system_test.ubuntu.plugins.thread_keystone_separate_services/125/artifact/logs/fail_error_separate_keystone_service-fuel-snapshot-2016-02-03_01-58-07.tar.xz.filtered.log

[Matthew Mosesohn (raytrac3r)]

openrc is fixed, but I'm still seeing timeouts on openstack token issue. It should, however, not be timing out on idempotent tasks (ones not containing 'create remove delete' in the action).

[Vladimir Kuklin (vkuklin)]

So far we figured out that node-1 uses memcached servers located on the controllers which are not yet set up, that's why it fails. We need to check why our memcached servers list is broken in the plugin and fix it.

[Dmitry Pyzhov (dpyzhov)]

It will be fixed in plugin

[Matthew Mosesohn (raytrac3r)]

https://review.openstack.org/#/c/275781/ and https://review.openstack.org/#/c/275791/ are on review.

[Matthew Mosesohn (raytrac3r)]

https://review.openstack.org/#/c/261547/ blocks this from being fixed

[Vladimir Khlyunev (vkhlyunev)]

The issue still exisis - ISO 536
Snapshot https://product-ci.infra.mirantis.net/view/8.0_swarm/job/8.0.system_test.ubuntu.plugins.thread_keystone_separate_services/130/artifact/logs/fail_error_separate_keystone_service-fuel-snapshot-2016-02-08_04-09-00.tar.xz
from node-4:/var/log/puppet.log

2016-02-08 02:45:48 +0000 Puppet (err): Command: 'openstack ["endpoint", "create", "--format", "shell", ["keystone", "--region", "RegionOne", "--publicurl", "https://public.fuel.local:5000/v2.0", "--internalurl", "http://10.109.11.4:5000/v2.0", "--adminurl", "http://10.109.11.4:35357/v2.0"]]' has been running for more then 20 seconds!

[Matthew Mosesohn (raytrac3r)]

We're still seeing failures related to mysql hash lookups. Review is here: https://review.openstack.org/#/c/276769/1

[Matthew Mosesohn (raytrac3r)]

and 8.0 backport: https://review.openstack.org/#/c/277458/

[slava valyavskiy (slava-val-al)]

Hello,

https://patching-ci.infra.mirantis.net/view/All/job/7.0.system_test.ubuntu.plugins.thread_db_separate_services

Task has been failed:
Task name - "database_hiera_override.pp"
2016-02-04 11:29:36 +0000 Puppet (err): Could not set 'file' on ensure: No such file or directory - /etc/hiera/plugins/detach-database.yaml20160204-15452-nh6z5p.lock at 112:/etc/fuel/plugins/detach-database-1.1/database_hiera_override.pp

[Denis Meltsaykin (dmeltsaykin)]

If the fix introduces regression in 7.0 tests, what should be done? Is it expected behavior and plugins do not support the older releases?

[Vadim Rovachev (vrovachev)]

Matthew, Hello.
Your patches https://review.openstack.org/#/c/275791/ https://review.openstack.org/#/c/275781/ breaks backward compatibility for 7.0 Release.

[Matthew Mosesohn (raytrac3r)]

7.0 was broken a long time ago with setup_repositories task addition and several others. I will make a separate maintenance branch for 7.0 after Fuel 8.0 release. I'm aware that it doesn't work and this is not the first backward incompatible patch.

[Matthew Mosesohn (raytrac3r)]

8.0.custom_system_test job #1800 passed. Let's see if tonight's swarm passes as well and then mark this resolved.

[ElenaRossokhina (esolomina)]

Verified:
Test passes since CI job #137
https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.plugins.thread_2_separate_services/137/testReport/%28root%29/separate_all_service/separate_all_service/

[Dmitriy Kruglov (dkruglov)]

Verified on CI against MOS 9.0, ISO build #519 - https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.plugins.thread_2_separate_services/152/.

[Alexey Stupnikov (astupnikov)]

We no longer support MOS5.1, MOS6.0, MOS6.1
We deliver only Critical/Security fixes to MOS7.0, MOS8.0.
We deliver only High/Critical/Security fixes to MOS9.2.