Improper configuration of OpenRC file leads to detached services error

[Tatyanka]

Deploy cluster with 3 separate keystone roles

Scenario:
1. Create cluster
2. Add 3 nodes with controller role
3. Add 3 nodes with keystone role
4. Add 1 compute and cinder
5. Verify networks
6. Deploy the cluster
7. Verify networks
8. Run OSTF

Actual REsult:
Deployment failed with:
http://paste.openstack.org/show/483720/

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "429"
  build_id: "429"
  fuel-nailgun_sha: "12b15b2351e250af41cc0b10d63a50c198fe77d8"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "df16d41cd7a9445cf82ad9fd8f0d53824711fcd8"
  fuel-nailgun-agent_sha: "92ebd5ade6fab60897761bfa084aefc320bff246"
  astute_sha: "c7ca63a49216744e0bfdfff5cb527556aad2e2a5"
  fuel-library_sha: "3eaf4f4a9b88b287a10cc19e9ce6a62298cc4013"
  fuel-ostf_sha: "214e794835acc7aa0c1c5de936e93696a90bb57a"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "85de57080a18fda18e5325f06eaf654b1b931592"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "e8e36cff332644576d7853c80b8a53d5b955420a"

[Timur Nurlygayanov (tnurlygayanov)]

SWARM tests in job https://product-ci.infra.mirantis.net/view/8.0_swarm/job/8.0.system_test.ubuntu.plugins.thread_keystone_separate_services/103/ failed during the deployment of the cluster with the following errors in puppet (please see the diagnostic snapshot for more detailed information):

 (/Stage[main]/Cinder::Keystone::Auth/Keystone::Resource::Service_identity[cinder]/Keystone_user[cinder]/ensure) change from absent to present failed: Execution of '/usr/bin/openstack domain list --quiet --format csv' returned 1:

 Execution of '/usr/bin/openstack service create --format shell volume --name cinder --description Cinder Service' returned 1:

Could not prefetch keystone_service provider 'openstack': Execution of '/usr/bin/openstack service list --quiet --format csv --long' returned 1:

[Matthew Mosesohn (raytrac3r)]

Vladimir Kuklin discovered that this is being caused by the following scenario:
1 - First controller (or keystone node) deploys okay
2 - Second node fails because apache is running and listening, HAProxy thinks Keystone is running (but it's not), and prefetch fails because the keystone service isn't running on that host yet.
3 - sometimes it succeeds on retries

We should improve the haproxy keystone check to see if we get 200 result on http://$ip:5000/v3/ request

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/267525

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/267528

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/267525
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=6bce1626a869e792e3077bcfc4a32bcc5dcb1453
Submitter: Jenkins
Branch: master

commit 6bce1626a869e792e3077bcfc4a32bcc5dcb1453
Author: Vladimir Kuklin <email address hidden>
Date: Thu Jan 14 15:46:42 2016 +0300

    Change URL for keystone backend check

    Set httpchk option to check /v3 URL in order
    to avoid getting false-positives from apache2
    in case when apache is listening but keystone
    is not yet configured.

    Change-Id: If78e7664c0184ff1064d80bd523c44156c005d69
    Closes-bug: #1533588

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/267528
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=3dedc561243a80a9dfc4276dd214d083dd7324d1
Submitter: Jenkins
Branch: stable/8.0

commit 3dedc561243a80a9dfc4276dd214d083dd7324d1
Author: Vladimir Kuklin <email address hidden>
Date: Thu Jan 14 15:46:42 2016 +0300

    Change URL for keystone backend check

    Set httpchk option to check /v3 URL in order
    to avoid getting false-positives from apache2
    in case when apache is listening but keystone
    is not yet configured.

    Change-Id: If78e7664c0184ff1064d80bd523c44156c005d69
    Closes-bug: #1533588
    (cherry picked from commit 6bce1626a869e792e3077bcfc4a32bcc5dcb1453)

[Dmitriy Kruglov (dkruglov)]

Verified on MOS 8.0, build 481.

The initial scenario is reproduced. The deployment is failed with the following error - http://paste.openstack.org/show/485269/.

ISO details:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "481"
  build_id: "481"
  fuel-nailgun_sha: "ae949905142507f2cb446071783731468f34a572"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "481ed135de2cb5060cac3795428625befdd1d814"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "420c6fa5f8cb51f3322d95113f783967bde9836e"
  fuel-ostf_sha: "ab5fd151fc6c1aa0b35bc2023631b1f4836ecd61"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "fac143f4dfa75785758e72afbdc029693e94ff2b"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "6c6b088a3d52dd0eaf43d59f3a3a149c93a07e7e"

[Matthew Mosesohn (raytrac3r)]

This bug fixes 404 errors caused by the secondary node deployment where haproxy is up but not ready to serve keystone. We're tracking issues caused by intermittent keystone availability in https://bugs.launchpad.net/fuel/+bug/1539117 instead

[Tatyanka (tatyana-leontovich)]

Reproduced on latest run https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.plugins.thread_keystone_separate_services/120/artifact/logs/fail_error_separate_keystone_service-fuel-snapshot-2016-01-28_22-43-52.tar.xz.filtered.log

[Vladimir Kuklin (vkuklin)]

Looking through the logs I cannot find any node that actually runs haproxy backend for keystone nodes. There is 0 nodes which executed openstack-haproxy-keystone task

[Vladimir Kuklin (vkuklin)]

okay, I found the RC

This is the config of openrc on node-1

#!/bin/sh
export LC_ALL=C
export OS_NO_CACHE='true'
export OS_TENANT_NAME='separatekeystone'
export OS_PROJECT_NAME='separatekeystone'
export OS_USERNAME='separatekeystone'
export OS_PASSWORD='separatekeystone'
export OS_AUTH_URL='/v2.0/'
export OS_DEFAULT_DOMAIN='default'
export OS_AUTH_STRATEGY='keystone'
export OS_REGION_NAME='RegionOne'
export CINDER_ENDPOINT_TYPE='internalURL'
export GLANCE_ENDPOINT_TYPE='internalURL'
export KEYSTONE_ENDPOINT_TYPE='internalURL'
export NOVA_ENDPOINT_TYPE='internalURL'
export NEUTRON_ENDPOINT_TYPE='internalURL'
export OS_ENDPOINT_TYPE='internalURL'
export MURANO_REPO_URL='http://storage.apps.openstack.org/'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/274196

[Vladimir Kuklin (vkuklin)]

Patch to the plugin as well https://review.openstack.org/274199

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/274894

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/274196
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=a20f59fbd01a326098c7dff21d636843933a2e52
Submitter: Jenkins
Branch: master

commit a20f59fbd01a326098c7dff21d636843933a2e52
Author: Vladimir Kuklin <email address hidden>
Date: Fri Jan 29 21:32:22 2016 +0300

    Switch auth_file URL generation to work properly

    This fix actually passes a variable to openstack::auth_file
    class which is using it then to interpolate it into
    a string in openrc file. Otherwise we have an internal_url
    which is improperly dereferenced and does not contain
    suffix for the URI to choose the version

    Change-Id: Ic3a70137e95df91c8e8fa257f16274e68075c800
    Partial-bug: #1533588
    Partial-bug: #1539644

[Matthew Mosesohn (raytrac3r)]

This will be fixed in time for 8.0 HCF

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/274894
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=da778978c9db257bf9d964d63679b0b3b50416b7
Submitter: Jenkins
Branch: stable/8.0

commit da778978c9db257bf9d964d63679b0b3b50416b7
Author: Vladimir Kuklin <email address hidden>
Date: Fri Jan 29 21:32:22 2016 +0300

    Switch auth_file URL generation to work properly

    This fix actually passes a variable to openstack::auth_file
    class which is using it then to interpolate it into
    a string in openrc file. Otherwise we have an internal_url
    which is improperly dereferenced and does not contain
    suffix for the URI to choose the version

    Partial-bug: #1533588
    Partial-bug: #1539644

    Change-Id: Ic3a70137e95df91c8e8fa257f16274e68075c800

[Dmitriy Kruglov (dkruglov)]

Verified on MOS 8.0, build 517. The initial error is not reproduced, so can be considered as fixed.
Yet the deployment fails with new symptoms and this one is tracked under https://bugs.launchpad.net/fuel/+bug/1540388.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "517"
  build_id: "517"
  fuel-nailgun_sha: "ebede8c0efab72b6e80735eb3a8161f953d84b83"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "601a0378f3e7f42ee2bc8f3d5f54fc681b6c5102"
  fuel-ostf_sha: "5fe41945c2a49f26c849df1fd46329f6db1ab6b0"
  fuel-mirror_sha: "6f8e1a005446eb49981fe93bd9a67ca944c0a763"
  fuelmenu_sha: "12227354aec1d38f7f51042df64cca59fa7a95f1"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "a365f05b903368225da3fea9aa42afc1d50dc9b4"

[Alex Schultz (alex-schultz)]

Verified on 9 #465