Keystone error Duplicate entry 'heat' for key 'ixu_domain_name'

[Egor Kotko]

Exec[heat_domain_create]/returns) change from notrun to 0 failed: heat-keystone-setup-domain returned 1 instead of one of [0]

http://jenkins-product.srt.mirantis.net:8080/job/7.0.system_test.ubuntu.thread_7/58/testReport/%28root%29/deploy_ceph_ha_nodegroups/deploy_ceph_ha_nodegroups/?

Scenario:
            1. Revert snapshot with 2 networks sets for slaves
            2. Create cluster (HA) with Neutron VXLAN and Ceph
            3. Add 3 controller + ceph nodes from default nodegroup
            4. Add 2 compute + ceph nodes from custom nodegroup
            5. Deploy cluster
            6. Deployment has failed. OpenStack cloud is not operational. In the puppet.log there should be the following error:

"Exec[heat_domain_create]/returns) change from notrun to 0 failed: heat-keystone-setup-domain returned 1 instead of one of [0]"

and the following error should be in keystone-all.log:

"Deploy has failed. In the puppet.log there should be the following error:"

{"build_id": "2015-08-10_17-24-26", "build_number": "156", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "2015-08-10_17-24-26", "build_number": "156", "api": "1.0", "fuel-library_sha": "c4b2b9a1b464d7f538635cbfc9304a714ededc41", "nailgun_sha": "9a6ac9b08733ded67bc65345a37787886ff4249d", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "57145b1d8804389304cd04322ba0fb3dc9d30327", "production": "docker", "python-fuelclient_sha": "ec5c02b3848fe1d15e5b50b323c3dda030f8e1ac", "astute_sha": "e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc", "fuel-ostf_sha": "c7f745431aa3c147f2491c865e029e0ffea91c47", "release": "7.0", "fuelmain_sha": "826387d1a05a40e4849433598442024bb1e3b7c7"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "c4b2b9a1b464d7f538635cbfc9304a714ededc41", "nailgun_sha": "9a6ac9b08733ded67bc65345a37787886ff4249d", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "57145b1d8804389304cd04322ba0fb3dc9d30327", "production": "docker", "python-fuelclient_sha": "ec5c02b3848fe1d15e5b50b323c3dda030f8e1ac", "astute_sha": "e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc", "fuel-ostf_sha": "c7f745431aa3c147f2491c865e029e0ffea91c47", "release": "7.0", "fuelmain_sha": "826387d1a05a40e4849433598442024bb1e3b7c7"}

[Artem Panchenko (apanchenko-8)]

Raising priority to 'Critical' because it causes BVT failure:

http://jenkins-product.srt.mirantis.net:8080/job/7.0.ubuntu.smoke_neutron/152/

[Vasyl Saienko (vsaienko)]

when heat-keystone-setup-domain was called, apache is restarting
https://paste.mirantis.net/show/904/
https://paste.mirantis.net/show/903/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/212412

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vasyl Saienko (<email address hidden>) on branch: master
Review: https://review.openstack.org/212412

[Vasyl Saienko (vsaienko)]

This bug is related https://bugs.launchpad.net/fuel/+bug/1472675

[Bogdan Dobrelya (bogdando)]

@Vasyl, what is the status of this bug and what is the patchset in progress, please link.

[Vasyl Saienko (vsaienko)]

@Bogdan: https://review.openstack.org/212439

[Bogdan Dobrelya (bogdando)]

It seems, the RC is heat specific errors:
 2015-08-11T00:21:28.246450+00:00 node-1 admin debug: 2015-08-11 00:21:28.246 18993 DEBUG keystone.common.sql.core [-] Conflict domain: (IntegrityError) (1062, "Duplicate entry 'heat' for key 'ixu_domain_name'") 'INSERT INTO domain (id, name, enabled, extra) VALUES (%s, %s, %s, %s)' ('e4d6984105764cbf8ff98e5830d6f1b4', 'heat', 1, '{"description": "Contains users and projects created by heat"}') wrapper /usr/lib/python2.7/dist-packages/keystone/common/sql/core.py:408
 2015-08-11T00:21:28.247210+00:00 node-1 admin warning: 2015-08-11 00:21:28.247 18993 WARNING keystone.common.wsgi [-] Conflict occurred attempting to store domain - Duplicate Entry

see the snippet: http://paste.openstack.org/show/2A7tfg0qDEjQozVSy6Cm/

[Bogdan Dobrelya (bogdando)]

Sorry, that was probably a keystone specific errors...

[Boris Bobrov (bbobrov)]

> admin warning: 2015-08-11 00:22:25.511 18993 WARNING keystone.common.wsgi [-] Conflict occurred attempting to store domain - Duplicate Entry

The log entry clearly says what the problem is -- the domain with this name already exists in the database.

[Bogdan Dobrelya (bogdando)]

From puppet side, it is also not clear why Service[httpd] recieved two restarts instead of one http://paste.openstack.org/show/8YrAiNjb8f2Tof5V46sx/
And as you can see, there were two more reloads or restarts with unknown source.
Probably, multiple restarts of admin/main services managed to corrupt keystone DB somehow

[Bogdan Dobrelya (bogdando)]

@Boris, thank you. I will consider your comment as an Invalid state for MOS project. Are you sure, this situation wasn't related with keystone app races?

[Bogdan Dobrelya (bogdando)]

JFYI, the same Duplicate entry error could be found here https://bugs.launchpad.net/mos/+bug/1446182, and keystone was running with eventlet. That bug was invalidated w/o proper RCA, by timeot. So I believe we 1) just faced it once again 2) It is not related with keystone under wsgi

[Boris Bobrov (bbobrov)]

> keystone app races

This can only happen if there are races in mysql claster. `ixu_domain_name` is an index which says that `name` is has a unique constraint.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/209924
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=ff2274e44ab732d943976d1b21c8997dc90a7d94
Submitter: Jenkins
Branch: master

commit ff2274e44ab732d943976d1b21c8997dc90a7d94
Author: vsaienko <email address hidden>
Date: Fri Aug 14 11:45:12 2015 +0300

    Tune tweak::apache_wrappers module

    - Sometimes apache fails to start after stop, due to unclosed
      resources. The problem frequently reproduced with keystone wsgi
      module, and didn't reproduced with horizon or radosgw.
      'apachectl restart' is recommended if doing start/stop rapidly
      https://wiki.apache.org/httpd/CouldNotBindToAddress
    - Redefine restart => 'apachectl graceful' for apache service
    - Remove disabling of GarbageCollector

    Related-Bug: #1472675
    Related-Bug: #1484066
    Related-Bug: #1457893
    Related-Bug: #1459357

    Change-Id: I34843639eacc9bcb6d451d3376440c8bfe9014f7

[Dmitry Tyzhnenko (dtyzhnenko)]

Verified on CI - https://product-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.thread_7/79/testReport/(root)/deploy_ceph_ha_nodegroups/

Fuel 7.0-262

{
 "build_id": "262",
 "build_number": "262",
 "auth_required": true,
 "fuel-ostf_sha": "582a81ccaa1e439a3aec4b8b8f6994735de840f4",
 "fuel-library_sha": "1556601b9b7503285714d7d1e02cc0807b1c68f0",
 "nailgun_sha": "b564ae20116297750bf6402b3a017e219bf4b468",
 "openstack_version": "2015.1.0-7.0",
 "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd",
 "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d",
 "api": "1.0",
 "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512",
 "astute_sha": "e63709d16bd4c1949bef820ac336c9393c040d25",
 "fuelmain_sha": "4dc6799370da4cddf06c04e4ecb7646102298535",
 "feature_groups": [
  "mirantis"
 ],
 "release": "7.0",
 "release_versions": {
  "2015.1.0-7.0": {
   "VERSION": {
    "build_id": "262",
    "build_number": "262",
    "fuel-library_sha": "1556601b9b7503285714d7d1e02cc0807b1c68f0",
    "nailgun_sha": "b564ae20116297750bf6402b3a017e219bf4b468",
    "fuel-ostf_sha": "582a81ccaa1e439a3aec4b8b8f6994735de840f4",
    "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd",
    "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d",
    "api": "1.0",
    "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512",
    "astute_sha": "e63709d16bd4c1949bef820ac336c9393c040d25",
    "fuelmain_sha": "4dc6799370da4cddf06c04e4ecb7646102298535",
    "feature_groups": [
     "mirantis"
    ],
    "release": "7.0",
    "openstack_version": "2015.1.0-7.0",
    "production": "docker"
   }
  }
 },
 "production": "docker"
}

[Vasyl Saienko (vsaienko)]

still reproduced on system tests:
http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/7.0.custom_system_test/763/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/224463

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/225447

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/224463
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=dc04a9c731291a6d5875db12810198eab2c841d3
Submitter: Jenkins
Branch: stable/7.0

commit dc04a9c731291a6d5875db12810198eab2c841d3
Author: vsaienko <email address hidden>
Date: Thu Sep 17 10:28:44 2015 +0300

    Create Heat Domain with Keystone_domain resource

     This patch replaces the usage of Exec to create the Heat domain, by
     using the Keystone_domain resource recently implemented in
     puppet-keystone.

    cherry picked from Upstream commit: 3ced80f)

    Change-Id: I8cd5af772867dfe00a131b119e12800a8e2ec021
    Closes-Bug: #1484066

[Davanum Srinivas (DIMS) (dims-v)]

fyi, custom_iso 1230 passed custom_bvt2 http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/7.0.custom.ubuntu.bvt_2/338/

[Davanum Srinivas (DIMS) (dims-v)]

So https://review.openstack.org/#/c/224463/ was merged

[Matthew Mosesohn (raytrac3r)]

reproduced on 8.0 in https://product-ci.infra.mirantis.net/view/8.0-kilo/job/8.0-kilo.ubuntu.bvt_2/
What is our timeline for addressing this in 8.0 either through upstream sync or cherry picking https://review.openstack.org/#/c/224463/ to 8.0?

[Vasyl Saienko (vsaienko)]

@Matthew: cherry pick is on review: https://review.openstack.org/#/c/225447/

[Vitaly Sedelnik (vsedelnik)]

As requested here is the summary of the impact and status of this issue:

Problem: Sporadically the deployment fails with error "Exec[heat_domain_create]/returns) change from notrun to 0 failed: heat-keystone-setup-domain returned 1 instead of one of [0]". OpenStack cloud is not functional.

Reproduction ratio: low (less than 1%) - per Vasyl we saw this issue ~10 times during 6.1 and 7.0 release cycles.

Workaround: Redeploy the cluster - most likely it will succeed because of low reproduction ratio.

RCA: Heat uses custom script [1] to setup keystone domain. It expects exception raised if domain already exists but sometimes the exception is not raised which leads to this issue - it tries to create duplicate record in the database. The reason why the exception is missing is still not clear - it's hard to debug the issue because of low reproduction ratio. We were unable to reproduce this issue manually.

Proposed fix: All services except Heat are using OpenStack client to create keystone domains. So the fix implements the domain creation in common way.

[1] https://github.com/openstack/heat/blob/master/bin/heat-keystone-setup-domain

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/225447
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=dc0aba4bd0e0f397b1090d9a7c5fdf106fafbce0
Submitter: Jenkins
Branch: master

commit dc0aba4bd0e0f397b1090d9a7c5fdf106fafbce0
Author: vsaienko <email address hidden>
Date: Thu Sep 17 10:28:44 2015 +0300

    Create Heat Domain with Keystone_domain resource

     This patch replaces the usage of Exec to create the Heat domain, by
     using the Keystone_domain resource recently implemented in
     puppet-keystone.

    cherry picked from Upstream commit: 3ced80f)

    Change-Id: I8cd5af772867dfe00a131b119e12800a8e2ec021
    Closes-Bug: #1484066

[Nastya Urlapova (aurlapova)]

Vitaliy, small correction about Reproduction ratio. Issue with heat was yesterday on bvt test, https://product-ci.infra.mirantis.net/view/8.0-kilo/job/8.0-kilo.ubuntu.bvt_2/52/
so I would say, that ration is higher then 1%.

[Vasyl Saienko (vsaienko)]

@Nastya: I don't agree with you, we launched 52 jobs and failed only 1.

[Vasyl Saienko (vsaienko)]

Also on old jenkins http://jenkins-product.srt.mirantis.net:8080/view/7.0/job/7.0.ubuntu.bvt_2/ 58 jobs were launched and 0 jobs were affected by this issue.
On new jenkins https://product-ci.infra.mirantis.net/job/7.0.ubuntu.bvt_2/ 49 jobs were launched and 0 jobs failed
On new jenkins 8.0 https://product-ci.infra.mirantis.net/view/8.0-kilo/job/8.0-kilo.ubuntu.bvt_2/ 49 jobs were launched and 1 jobs failed

Total: 156 jobs were launched and failed only 1.

[Nastya Urlapova (aurlapova)]

Vasiliy, original description and comment #2 might have escaped your attention.

[Tatyanka (tatyana-leontovich)]

for 7.0 verified {"build_id": "301", "build_number": "301", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "301", "build_number": "301", "api": "1.0", "fuel-library_sha": "5d50055aeca1dd0dc53b43825dc4c8f7780be9dd", "nailgun_sha": "4162b0c15adb425b37608c787944d1983f543aa8", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "50e90af6e3d560e9085ff71d2950cfbcca91af67", "production": "docker", "python-fuelclient_sha": "486bde57cda1badb68f915f66c61b544108606f3", "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012", "fuel-ostf_sha": "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c", "release": "7.0", "fuelmain_sha": "a65d453215edb0284a2e4761be7a156bb5627677"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "5d50055aeca1dd0dc53b43825dc4c8f7780be9dd", "nailgun_sha": "4162b0c15adb425b37608c787944d1983f543aa8", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "50e90af6e3d560e9085ff71d2950cfbcca91af67", "production": "docker", "python-fuelclient_sha": "486bde57cda1badb68f915f66c61b544108606f3", "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012", "fuel-ostf_sha": "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c", "release": "7.0", "fuelmain_sha": "a65d453215edb0284a2e4761be7a156bb5627677"}

[Dmitriy Kruglov (dkruglov)]

Verified on 8.0 kilo. The issue is not reproduced.

ISO info:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "108"
  build_id: "108"
  fuel-nailgun_sha: "ec56b54491ecb0182f66e675f549825abc1a96ee"
  python-fuelclient_sha: "286939d3be220828f52e73b65928ed39662e1853"
  fuel-agent_sha: "e881f0dabd09af4be4f3e22768b02fe76278e20e"
  fuel-nailgun-agent_sha: "d66f188a1832a9c23b04884a14ef00fc5605ec6d"
  astute_sha: "0f753467a3f16e4d46e7e9f1979905fb178e4d5b"
  fuel-library_sha: "acfcfd289ca454585687b6ff9651b53e4ffaf0cd"
  fuel-ostf_sha: "37c5d6113408a29cabe0f416fe99cf20e2bca318"
  fuelmain_sha: "8e5e75302b2534fd38e4b41b795957111ac75543"