Neutron HA cluster deployment failed with (/Stage[main]/Neutron::Server::Notifications/Nova_admin_tenant_id_setter[nova_admin_tenant_id]/ensure) change from absent to present failed: execution expired

[Andrey Sledzinskiy]

{

    "build_id": "2014-11-03_11-06-52",
    "ostf_sha": "9c6fadca272427bb933bc459e14bb1bad7f614aa",
    "build_number": "56",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "35946b1f225c984f11915ba8e985584160f0b129",
    "production": "docker",
    "fuelmain_sha": "9012d6252d23f56a244142e5341bdcb2fc1ce5ef",
    "astute_sha": "c72dac7b31646fbedbfc56a2a87676c6d5713fcf",
    "feature_groups": [
        "mirantis"
    ],
    "release": "6.0",
    "release_versions": {
        "2014.2-6.0": {
            "VERSION": {
                "build_id": "2014-11-03_11-06-52",
                "ostf_sha": "9c6fadca272427bb933bc459e14bb1bad7f614aa",
                "build_number": "56",
                "api": "1.0",
                "nailgun_sha": "35946b1f225c984f11915ba8e985584160f0b129",
                "production": "docker",
                "fuelmain_sha": "9012d6252d23f56a244142e5341bdcb2fc1ce5ef",
                "astute_sha": "c72dac7b31646fbedbfc56a2a87676c6d5713fcf",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "6.0",
                "fuellib_sha": "45ad9b42666d7e3e14ab9af2911808e6c8806842"
            }
        }
    },
    "fuellib_sha": "45ad9b42666d7e3e14ab9af2911808e6c8806842"

}

Steps:
1. Create next cluster - CentOS, HA, Neutron Vlan, 3 controllers, 2 compute nodes
2. Deploy cluster

Actual result - deployment failed with errors in puppet logs on node-2
2014-11-03 13:40:51 ERR

 Could not run: execution expired

2014-11-03 13:40:50 ERR

 (/Stage[main]/Neutron::Server::Notifications/Nova_admin_tenant_id_setter[nova_admin_tenant_id]/ensure) change from absent to present failed: execution expired

2014-11-03T13:40:50.597795+00:00 debug: Storing state
2014-11-03T13:40:50.598728+00:00 info: Creating state file /var/lib/puppet/state/state.yaml
2014-11-03T13:40:51.004546+00:00 debug: Stored state in 0.41 seconds
2014-11-03T13:40:51.020320+00:00 err: Could not run: execution expired
2014-11-03T13:40:51.022231+00:00 err: /usr/lib/ruby/1.8/timeout.rb:64:in `rbuf_fill'
2014-11-03T13:40:51.022231+00:00 err: /usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'

Logs are attached

[Xav Paice (xavpaice)]

Added puppet-neutron, because I'm seeing the same error without Fuel, using Ubuntu 12.04 and the current stable/Icehouse branch of puppet-neutron.

If/when I find a fix, will update.

[Dmitry Borodaenko (angdraug)]

Xav, please comment which exactly error you're getting from Neutron. Have you found a neutron stack trace in the attached logs bundle that matches what you're getting without Fuel? All we currently have in the bug description is a puppet error, not sure how it can be useful to neutron developers.

Fuel QA: if this is confirmed to be a neutron bug, we need to do the following:
1) move the bug from fuel to mos project
2) assign mos-neutron
3) create a separate bug against neutron and remove neutron from this bug (we should avoid spamming upstream teams with Fuel/MOS specific bug status updates)

[Stanislav Makar (smakar)]

I have just deployed such type of env(Juno on CentOS , neutron vlan ha) using the iso fuel-6.0-75-2014-11-04_16-38-46.iso
All is fine
Andrey, where did you get this ISO?
I could not find it, especially #56 (2014-11-03_11-06-52)

[Andrey Sledzinskiy (asledzinskiy)]

Stanislav, it's iso from /job/6.0.staging.iso/ tab

[Sergey Vasilenko (xenolog)]

It's a not neutron-related bug, besause:

Mon Nov 03 14:05:50 +0000 2014 /Stage[main]/Neutron::Server::Notifications/Nova_admin_tenant_id_setter[nova_admin_tenant_id]/ensure (err): change from absent to present failed: Received error response from Keystone server at http://10.108.2.2:35357/v2.0/tokens: Gateway Time-out
Mon Nov 03 14:05:50 +0000 2014 /Stage[main]/Neutron::Server::Notifications/Nova_admin_tenant_id_setter[nova_admin_tenant_id] (debug): Nothing to manage: no ensure and the resource doesn't exist
Mon Nov 03 14:05:50 +0000 2014 /Stage[main]/Neutron::Server::Notifications/Nova_admin_tenant_id_setter[nova_admin_tenant_id] (info): Evaluated in 60.37 seconds

Keystone didn't answer to the request long time.

[Aleksandr Didenko (adidenko)]

I agree with Sergey - it's Keystone issue, see the logs:

2014-11-03T13:40:50.597669+00:00 err: (/Stage[main]/Neutron::Server::Notifications/Nova_admin_tenant_id_setter[nova_admin_tenant_id]/ensure) change from absent to present failed: execution expired

2014-11-03T13:40:50.596700+00:00 info: 10.108.2.4:35687 [03/Nov/2014:13:39:50.571] keystone-2 keystone-2/node-2 0/0/1/-1/60001 504 194 - - sH-- 2/0/0/0/0 0/0 "POST /v2.0/tokens HTTP/1.1"

Unfortunately due to https://bugs.launchpad.net/fuel/+bug/1381545 there are no keystone logs in the snapshot. So please provide a live env with this error if you get it again.

[Aleksandr Didenko (adidenko)]

It also may be a duplicate of https://bugs.launchpad.net/fuel/+bug/1376686
But we need a live env to ensure.

[Xav Paice (xavpaice)]

Tested again on a fresh build - I marked this for puppet-neutron because it was a puppet error in the neutron deployment stages - however I'm finding this time round (Icehouse on precise) it's absolutely fine and I cannot reproduce the error. The logs I was seeing were the same as Sergey and Aleksandr point out.

[Vladimir Kuklin (vkuklin)]

I am pretty sure that this bug was related to https://bugs.launchpad.net/mos/+bug/1370324

[Anastasia Palkina (apalkina)]

Cannot reproduce on ISO #49

"build_id": "2014-12-09_22-41-06", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "49", "auth_required": true, "api": "1.0", "nailgun_sha": "22bd43b89a17843f9199f92d61fc86cb0f8772f1", "production": "docker", "fuelmain_sha": "3aab16667f47dd8384904e27f70f7a87ba15f4ee", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-09_22-41-06", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "49", "api": "1.0", "nailgun_sha": "22bd43b89a17843f9199f92d61fc86cb0f8772f1", "production": "docker", "fuelmain_sha": "3aab16667f47dd8384904e27f70f7a87ba15f4ee", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "2c99931072d951301d395ebd5bf45c8d401301bb"}}}, "fuellib_sha": "2c99931072d951301d395ebd5bf45c8d401301bb"