Detached keystone fails deployment with use public ssl error

[Alexander Kurenyshev]

Found on CI job https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.plugins.thread_2_separate_services/91/console

Deploy failed with error:
2015-12-30 02:16:30 ERROR [804] Task '{"priority"=>4800, "type"=>"puppet", "id"=>"keystone-controller", "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"keystone-controller.pp", "timeout"=>600, "cwd"=>"/etc/fuel/plugins/detach-keystone-1.0/"}, "uids"=>["1"]}' failed on node 1

At the puppet logs on node-1:

2015-12-30 02:16:28 +0000 Puppet (err): You must set up path to public ssl keypair if you want to use public ssl at /etc/puppet/modules/openstack/manifests/ha/haproxy_service.pp:103 on node node-1.test.domain.local

Fuel ISO #368

[Alexander Kurenyshev (akurenyshev)]

The same situation is here https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.plugins.thread_keystone_separate_services/88/console

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/264728

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/264728
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-detach-keystone/commit/?id=3c1ffde9ae649bd57bf7223ce83a07699e601665
Submitter: Jenkins
Branch: master

commit 3c1ffde9ae649bd57bf7223ce83a07699e601665
Author: Bartosz Kupidura <email address hidden>
Date: Thu Jan 7 12:15:06 2016 +0100

    Use SSL path when SSL enabled

    Change-Id: Id292e2411902f0e4f3ea86402ee58eb7d07bfd11
    Closes-Bug: #1530119

[Tatyanka (tatyana-leontovich)]

was merged in master and not backported into stable, so I set correct statuses

[Bartosz Kupidura (zynzel)]

In detach-keystone plugin we have only master branch. So there is no need for backport.

[Tatyanka (tatyana-leontovich)]

looks like issue still reproduces:
plugin used in tests http://jenkins-product.srt.mirantis.net:8080/job/build-fuel-plugins/lastSuccessfulBuild/artifact/built_plugins/detach-keystone-1.0-1.0.2-1.noarch.rpm
(contains changes), but tests failed with the same error
https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.plugins.thread_keystone_separate_services/102/testReport/%28root%29/separate_keystone_service/
You must set up path to public ssl keypair if you want to use public ssl at /etc/puppet/modules/openstack/manifests/ha/haproxy_service.pp:103 on node node-1.test.domain.local

[Bartosz Kupidura (zynzel)]

This env was deployed with old plugin version:
root@node-1:/etc/fuel/plugins/detach-keystone-1.0# grep -c 'path to public ssl keypair' /var/log/puppet.log
6
root@node-1:/etc/fuel/plugins/detach-keystone-1.0# grep 'public_ssl' /etc/fuel/plugins/detach-keystone-1.0/keystone-controller.pp
$public_ssl_hash = hiera('public_ssl')
  public_ssl => $public_ssl_hash['services'],

How this should look like:
131156:banshee.local$ grep 'public_ssl' keystone-controller.pp
$public_ssl_hash = hiera('public_ssl')
$public_ssl = get_ssl_property($ssl_hash, $public_ssl_hash, 'keystone', 'public', 'usage', false)
$public_ssl_path = get_ssl_property($ssl_hash, $public_ssl_hash, 'keystone', 'public', 'path', [''])
  public_ssl => $public_ssl,
  public_ssl_path => $public_ssl_path,

Please rerun test with laster master of detach-keystone plugin

[Timur Nurlygayanov (tnurlygayanov)]

I checked the latest job logs:
https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.plugins.thread_2_separate_services/105/console

and looks like SWARM tests for separate keystone node deployment passed in this run.
So, bug marked as Invalid because it was the issue with incorrect version of the plugin (it is not clear why we used incorrect version of the plugin it SWARM)

[Timur Nurlygayanov (tnurlygayanov)]

the original issue not reporduced, for the latest snapshot:

xwizard@DFldslr:~/fuel-snapshot-2016-01-12_07-18-39$ egrep -e "failed on node" -r ./
xwizard@DFldslr:~/fuel-snapshot-2016-01-12_07-18-39$ egrep -e "You must set up path to public ssl" -r ./
xwizard@DFldslr:~/fuel-snapshot-2016-01-12_07-18-39$

[Tatyanka (tatyana-leontovich)]

Timur actually job you are linked contains failed test by this reason, so move to incomplete and if you revert the env you'll see that plugin (rpm) contain the fix, so move back to incomplete

[Timur Nurlygayanov (tnurlygayanov)]

The latest diagnostic snapshot:
https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.plugins.thread_2_separate_services/105/artifact/logs/fail_error_separate_all_service-fuel-snapshot-2016-01-12_07-18-39.tar.xz

[Timur Nurlygayanov (tnurlygayanov)]

I reverted the latest snapshot and we can see the following on the node with separate keystone service:

root@node-3:/etc/fuel/plugins/detach-keystone-1.0# md5sum keystone-controller.pp
5035319a2ecd00d5f0aba1fafed83a8a keystone-controller.pp

and we can see that we have another version of this puppet manifest in http://jenkins-product.srt.mirantis.net:8080/job/build-fuel-plugins/lastSuccessfulBuild/artifact/built_plugins/:

xwizard@DFldslr:~/detach-keystone-1.0/deployment_scripts$ md5sum keystone-controller.pp
ee0eda41c3538e1c5e82c128b2bd1842 keystone-controller.pp

but on master node:
[root@nailgun deployment_scripts]# md5sum keystone-controller.pp
5035319a2ecd00d5f0aba1fafed83a8a keystone-controller.pp

[Timur Nurlygayanov (tnurlygayanov)]

So, the reason of the issue in the old package:
http://jenkins-product.srt.mirantis.net:8080/job/build-fuel-plugins/lastSuccessfulBuild/artifact/built_plugins/detach-keystone-1.0-1.0.2-1.noarch.rpm

we need to update this package and then tests should pass.

How to verify the fix:
md5 sum for this package should be different:
http://jenkins-product.srt.mirantis.net:8080/job/build-fuel-plugins/lastSuccessfulBuild/artifact/built_plugins/detach-keystone-1.0-1.0.2-1.noarch.rpm:

wget http://jenkins-product.srt.mirantis.net:8080/job/build-fuel-plugins/lastSuccessfulBuild/artifact/built_plugins/detach-keystone-1.0-1.0.2-1.noarch.rpm && md5sum detach-keystone-1.0-1.0.2-1.noarch.rpm

ced278983a71f6a7b4c8c5f8aae2bdc0 detach-keystone-1.0-1.0.2-1.noarch.rpm

^^ it is for old package.

[Timur Nurlygayanov (tnurlygayanov)]

from the github repository:
xwizard@DFldslr:~/detach-keystone-1.0$ md5sum keystone-controller.pp
ee0eda41c3538e1c5e82c128b2bd1842 keystone-controller.pp

from the latest jenkins job:
xwizard@DFldslr:~/fuel-plugin-detach-keystone/deployment_scripts$ md5sum keystone-controller.pp
ee0eda41c3538e1c5e82c128b2bd1842 keystone-controller.pp

it means that we have the latest version of the plugin in rpm package.

On the master node in test environment which was reverted from the latest snapshot:
[root@nailgun ~]# md5sum /var/www/nailgun/plugins/detach-keystone-1.0/deployment_scripts/keystone-controller.pp
ee0eda41c3538e1c5e82c128b2bd1842 /var/www/nailgun/plugins/detach-keystone-1.0/deployment_scripts/keystone-controller.pp

the version is the same.

[Timur Nurlygayanov (tnurlygayanov)]

but tests still fail

[Timur Nurlygayanov (tnurlygayanov)]

Fix verified on SWARM jobs execution.

Tests still fail but because of another issue:
https://bugs.launchpad.net/fuel/+bug/1533649