ERR: bash -c "source /root/openrc ; keystone tenant-list" | grep "services" > /tmp/services returned 1 instead of one of [0]

[Anastasia Palkina]

"build_id": "2014-06-10_00-30-36",
"mirantis": "yes",
"build_number": "249",
"ostf_sha": "67b61ed3788297fa5d985afec32498d8c0f812db",
"nailgun_sha": "7cccb4f0b91914bc3b176d1f2ee285ad6f21431f",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "e59cb325d99fba49085ed5b0b6cfb0d9a83d2fc8",
"astute_sha": "17b1afa5f0dc8f4fca5ed4eb03ec566fbfb5ed19",
"release": "5.1",
"fuellib_sha": "dd8ed9496870d64d931885298aa17100b9b78ddc"

1. Create new environment (Ubuntu, simple mode)
2. Choose GRE segmentation
3. Choose both Ceph
4. Choose Sahara and Ceilometer installation
5. Add controller+ceph, compute+ceph, mongo
6. Start deployment. It was successful
7. But there is error in puppet.log on controller (node-1):

2014-06-10 09:11:31 ERR

 bash -c "source /root/openrc ; keystone tenant-list" | grep "services" > /tmp/services returned 1 instead of one of [0]

[Anastasia Palkina (apalkina)]

Reproduced on ISO #250. Deployment has failed.

"build_id": "2014-06-11_00-30-36",
"mirantis": "yes",
"build_number": "250",
"ostf_sha": "67b61ed3788297fa5d985afec32498d8c0f812db",
"nailgun_sha": "f851199db57d9561b51c429956c056b463ad3f0b",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "e59cb325d99fba49085ed5b0b6cfb0d9a83d2fc8",
"astute_sha": "17b1afa5f0dc8f4fca5ed4eb03ec566fbfb5ed19",
"release": "5.1",
"fuellib_sha": "d9b8daa61e131e3c346e93d3870620897a71c593"

1. Create new environment (CentOS, HA mode)
2. Choose GRE segmentation
3. Choose Ceilometer
4. Add 3 controllers, 1 compute, 1 mongo
5. Start deployment. It has failed.

[Anastasia Palkina (apalkina)]

Controller node-6

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/100263

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/100263
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=4cc3679a4d2b3492638585a2af9677d223868316
Submitter: Jenkins
Branch: master

commit 4cc3679a4d2b3492638585a2af9677d223868316
Author: Sergey Vasilenko <email address hidden>
Date: Mon Jun 16 18:30:24 2014 +0400

    more strong order in Neutron manifests

    between keystone roles and inserting 'service' tenant-ID to the neutron config.

    Change-Id: I9d6780fe924756643f34a597a1abd9e6189611ee
    Closes-bug: #1328462

[Anastasia Palkina (apalkina)]

Verified on ISO #278
"build_id": "2014-06-30_00-31-14",
"mirantis": "yes",
"build_number": "278",
"ostf_sha": "7171951202a22d8dfe9b3b76c9824001e2485630",
"nailgun_sha": "bcb3f0795acbd21f4882103c735a64051492cf7f",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "74ee2ba18e29f81917227767bbf77dfc4c9ffc1d",
"astute_sha": "d1d54d57b3ce1b2f975cbc16c354f1ae7b3391ee",
"release": "5.1",
"fuellib_sha": "c6adc9f28e722f3845da49e45f1e1729d6f6b1a0"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/114189

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sergii Golovatiuk (<email address hidden>) on branch: stable/5.0
Review: https://review.openstack.org/114189

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/117290

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/117290
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=554f512cd86068194672648bf5c81128bdfd3294
Submitter: Jenkins
Branch: stable/5.0

commit 554f512cd86068194672648bf5c81128bdfd3294
Author: Sergey Vasilenko <email address hidden>
Date: Mon Jun 16 18:30:24 2014 +0400

    more strong order in Neutron manifests

    between keystone roles and inserting 'service' tenant-ID to the neutron config.

    Change-Id: I9d6780fe924756643f34a597a1abd9e6189611ee
    Closes-bug: #1328462

[Egor Kotko (ykotko)]

Got similar bug on:
{"build_id": "2014-11-11_20-08-59", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "33", "auth_required": true, "api": "1.0", "nailgun_sha": "bbc9dfe78a0c33040dcd16de9a40a3491788719c", "production": "docker", "fuelmain_sha": "88d4289e88e4bd88a3cabaf15a11ae8fc9ded53f", "astute_sha": "702af3db6f5bca92525bc8322d7d5d7675ec857e", "feature_groups": ["mirantis"], "release": "5.1.1", "release_versions": {"2014.1.1-5.1.1": {"VERSION": {"build_id": "2014-11-11_20-08-59", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "33", "api": "1.0", "nailgun_sha": "bbc9dfe78a0c33040dcd16de9a40a3491788719c", "production": "docker", "fuelmain_sha": "88d4289e88e4bd88a3cabaf15a11ae8fc9ded53f", "astute_sha": "702af3db6f5bca92525bc8322d7d5d7675ec857e", "feature_groups": ["mirantis"], "release": "5.1.1", "fuellib_sha": "e5b3de834a400d98d8c6ba416249832a0c16076c"}}}, "fuellib_sha": "e5b3de834a400d98d8c6ba416249832a0c16076c"}

Deploy cluster in HA mode with Neutron VLAN Centos

        Scenario:
            1. Create cluster
            2. Add 3 nodes with controller role
            3. Add 2 nodes with compute role
            4. Deploy the cluster
            5. Run network verification
            6. Run OSTF

Puppet error log from first controller:
http://paste.openstack.org/show/132399/

Syste test result:
http://jenkins-product.srt.mirantis.net:8080/view/5.1_swarm/job/5.1_fuelmain.system_test.centos.thread_3/45/testReport/%28root%29/deploy_neutron_vlan_ha/deploy_neutron_vlan_ha/
http://paste.openstack.org/show/132399/

[Sergey Vasilenko (xenolog)]

Debug snapshot has no neutron server log :(

but those lines:
Wed Nov 12 09:49:17 +0000 2014 Puppet (debug): Executing 'bash -c "source /root/openrc ; keystone tenant-list" | grep "services" > /tmp/services'
Wed Nov 12 09:50:17 +0000 2014 /Stage[main]/Neutron::Server/Exec[get_service_tenant_ID]/returns (debug): Sleeping for 3.0 seconds between tries
Wed Nov 12 09:50:20 +0000 2014 /Stage[main]/Neutron::Server/Exec[get_service_tenant_ID]/returns (notice): Authorization Failed: Gateway Timeout (HTTP 504)
Wed Nov 12 09:50:20 +0000 2014 Puppet (err): bash -c "source /root/openrc ; keystone tenant-list" | grep "services" > /tmp/services returned 1 instead of one of [0]

say about it's a keystone-related bug.

[Dmitry Borodaenko (angdraug)]

Looks like a generic "Keystone is broken" problem. Please do not resurrect old generic bugs, see an explanation here:
https://bugs.launchpad.net/fuel/+bug/1316524/comments/9