CentOS HA 3 controller cluster re-deployment failed after adding two new controllers with heat-keystone-setup-domain &>/dev/null returned 1 instead of one of [0]

[Andrey Sledzinskiy]

{"build_id": "2015-04-17_15-24-00",
"ostf_sha": "4bab9b975ace8d9a305d6e0f112b734de587f847",
"build_number": "321",
"release_versions": {
"2014.2-6.1": {
"VERSION": {
"build_id": "2015-04-17_15-24-00",
"ostf_sha": "4bab9b975ace8d9a305d6e0f112b734de587f847",
"build_number": "321",
"api": "1.0",
"nailgun_sha": "939e5780cd0f7b1af3afd2926eda30f81bfc3e3f",
"openstack_version": "2014.2-6.1",
"production": "docker",
"python-fuelclient_sha": "0698062e9b044becf07bf9918fa16613aa3d93ad",
"astute_sha": "bf1751a4fe0d912325e3b4af629126a59c1b2b51",
"feature_groups":
["mirantis"],
"release": "6.1",
"fuelmain_sha": "5981d230e9484c196022a027c5c1600e36b17a72",
"fuellib_sha": "65617981bef34ea96b85d2d389cc037c304516e5"}}},
"auth_required": true,
"api": "1.0",
"nailgun_sha": "939e5780cd0f7b1af3afd2926eda30f81bfc3e3f",
"openstack_version": "2014.2-6.1",
"production": "docker",
"python-fuelclient_sha": "0698062e9b044becf07bf9918fa16613aa3d93ad",
"astute_sha": "bf1751a4fe0d912325e3b4af629126a59c1b2b51",
"feature_groups":
["mirantis"],
"release": "6.1",
"fuelmain_sha": "5981d230e9484c196022a027c5c1600e36b17a72",
"fuellib_sha": "65617981bef34ea96b85d2d389cc037c304516e5"}

Steps:
1. Create next cluster - HA, CentOS, flat nova-network, 1 controller
2. Deploy it
3. Add 2 new controllers
4. Deploy cluster
5. Add 2 new controllers
6. Start deployment

Actual result - re-deployment failed with errors in puppet (node-1):
(/Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns) change from notrun to 0 failed: heat-keystone-setup-domain &>/dev/null returned 1 instead of one of [0]

Logs are attached

[Igor Yozhikov (iyozhikov)]

Unfortunately, I can't reproduce this bug due errors related to unavailability of dnsmasq running in namespace.

For now found next from log files:

puppet:
Mon Apr 20 07:57:03 +0000 2015 Puppet (debug): Executing 'heat-keystone-setup-domain &>/dev/null'
Mon Apr 20 07:57:05 +0000 2015 Puppet (err): heat-keystone-setup-domain &>/dev/null returned 1 instead of one of [0]
 keystone:
2015-04-20T07:57:04.869237+00:00 debug: Conflict user: (IntegrityError) (1062, "Duplicate entry '773c50c765654c34a327ebf5f648d451-heat_admin' for key 'ixu_user_name_domain_id'") 'INSERT INTO user (id, name, domain_id, password, enabled,
extra, default_project_id) VALUES (%s, %s, %s, %s, %s, %s, %s)' ('e2b5cdc441f8478eb1dc91478b74621f', 'heat_admin', '773c50c765654c34a327ebf5f648d451', '$6$rounds=40000$3cC3EP7SBFwIKN9h$8K/HlFccJBOCbIObN46T1IQQ71aMOsO.FzsRCX7HhS5RzbMVMT3u6
0usrWF7LoDXn9Vtg//m90maB9x8WEqPW/', 1, '{"description": "Heat domain admin"}', None)
2015-04-20T07:57:04.870658+00:00 warning: Conflict occurred attempting to store user - Duplicate Entry

puppet:
Mon Apr 20 07:51:52 +0000 2015 Puppet (debug): Executing 'heat-keystone-setup-domain &>/dev/null'
Mon Apr 20 07:51:56 +0000 2015 Puppet (err): heat-keystone-setup-domain &>/dev/null returned 1 instead of one of [0]
keystone:
2015-04-20T07:51:23.723210+00:00 debug: Conflict domain: (IntegrityError) (1062, "Duplicate entry 'heat' for key 'ixu_domain_name'") 'INSERT INTO domain (id, name, enabled, extra) VALUES (%s, %s, %s, %s)' ('1a41fc6a560848a6b258d5e5c204e5c3', 'heat', 1, '{"description": "Contains users and projects created by heat"}')
2015-04-20T07:51:23.725353+00:00 warning: Conflict occurred attempting to store domain - Duplicate Entry
2015-04-20T07:51:55.877181+00:00 debug: Conflict user: (IntegrityError) (1062, "Duplicate entry '773c50c765654c34a327ebf5f648d451-heat_admin' for key 'ixu_user_name_domain_id'") 'INSERT INTO user (id, name, domain_id, password, enabled, extra, default_project_id) VALUES (%s, %s, %s, %s, %s, %s, %s)' ('c540b767e7e04d639ae7fa039908f29d', 'heat_admin', '773c50c765654c34a327ebf5f648d451', '$6$rounds=40000$ySdzXk5bzbMovEbb$2iBcMbe2xWNtTGtqQnfLFqZdb/gGD3RMRB.9A2wvfRz1vGqm.Cn.Ril3/eloWQww9lt4QWFm2wgjo4F3LQJGA/', 1, '{"description": "Heat domain admin"}', None)
2015-04-20T07:51:55.906372+00:00 warning: Conflict occurred attempting to store user - Duplicate Entry

puppet:
Mon Apr 20 07:51:20 +0000 2015 Puppet (debug): Executing 'heat-keystone-setup-domain &>/dev/null'
Mon Apr 20 07:51:23 +0000 2015 Puppet (err): heat-keystone-setup-domain &>/dev/null returned 1 instead of one of [0]
keystone:
015-04-20T07:51:23.723210+00:00 debug: Conflict domain: (IntegrityError) (1062, "Duplicate entry 'heat' for key 'ixu_domain_name'") 'INSERT INTO domain (id, name, enabled, extra) VALUES (%s, %s, %s, %s)' ('1a41fc6a560848a6b258d5e5c204e5c3', 'heat', 1, '{"description": "Contains users and projects created by heat"}')
2015-04-20T07:51:23.725353+00:00 warning: Conflict occurred attempting to store domain - Duplicate Entry

[Oleksiy Molchanov (omolchanov)]

Moved to invalid, as this issue was not updated for more than 3 weeks.