Activity log for bug #1496840

Date Who What changed Old value New value Message
2015-09-17 12:41:58 Paul Karikh bug added bug
2015-09-17 12:42:33 Paul Karikh description I've deployed MOS 7.0 with Fuel Keystone LDAP plugin and tried to list users with Horizon. Everything works slow, but fine and there were no errors. After that I've created 5K users in the LDAP. Somewhere after 1.5K users Horizon requests to Keystone started to failing with timeout error. When Horizon is waiting for Keystone response, a I see in Keystone logs (/var/log/keystone/main.log) a lot of debug lines like this: `2015-09-17 11:37:47.990 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:47.991 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1067 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510` After that I've deleted almost all users from LDAP (there left only auto_user0, auto_user1 and auto_user10 and admin users) and tried to fetch them again with Horizon. And I've got the same lines: 2015-09-17 11:37:47.969 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510 2015-09-17 11:37:47.978 12575 DEBUG keystone.identity.core [-] Found existing mapping to public ID: a06df89598f65718695a3c3e138451f351416dfbd3f39fb254462f438de4c274 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:523 2015-09-17 11:37:47.980 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:47.982 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-10 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510 2015-09-17 11:37:47.989 12575 DEBUG keystone.identity.core [-] Found existing mapping to public ID: 44c396a392a341539101a28cdad4312d5113ca88c264fb4135550c0e336ae579 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:523 2015-09-17 11:37:47.990 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:47.991 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1067 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510 2015-09-17 11:37:47.998 12575 DEBUG keystone.identity.core [-] Found existing mapping to public ID: 85ca3743b8406ffd60505fa16342f66cf3f6b7ed9d8d197e3beab28e2715930c _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:523 2015-09-17 11:37:48.000 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:48.001 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1068 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510 2015-09-17 11:37:48.009 12575 DEBUG keystone.identity.core [-] Found existing mapping to public ID: 1de806c18b4cca49aefb0820e1097e8df5fd68ea0b1425b74643eacbc2e9d981 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:523 2015-09-17 11:37:48.011 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:48.012 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1069 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510 2015-09-17 11:37:48.019 12575 DEBUG keystone.identity.core [-] Found existing mapping to public ID: 94f6f8fbac83e4ae87fab63758322dabfaf43b9187a77f3b50d2bb66f26be5e4 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:523 2015-09-17 11:37:48.020 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:48.021 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1070 The problem is users auto_user-1070 , auto_user-1069, auto_user-1067, etc are no more present in LDAP. And even after 24 hours after users has been deleted from LDAP there are a lot of lines where Keystone says that it maps these users. And looks like while Keystone maps users, Horizon is waiting fore response. So, even if LDAP has not so much users, Horizon can get responce from Keystone and fails with timeout. Here is how LDAP is configured with Keystone: /etc/keystone/domains/keystone.keystone.tld.conf contains the followong lines [ldap] user_allow_update=False user=cn=admin_ad,cn=Users,dc=keystone,dc=tld user_filter= user_name_attribute=cn user_pass_attribute=userPassword user_enabled_attribute=enabled suffix=dc=keystone,dc=tld password=Pass1234 url=ldap://172.16.57.78 user_allow_create=False user_allow_delete=False user_objectclass=person user_tree_dn=dc=keystone,dc=tld query_scope=sub user_id_attribute=cn debug_level=-1 page_size = 50 [identity] driver=keystone.identity.backends.ldap.Identity I've deployed MOS 7.0 with Fuel Keystone LDAP plugin and tried to list users with Horizon. Everything works slow, but fine and there were no errors. After that I've created 5K users in the LDAP. Somewhere after 1.5K users Horizon requests to Keystone started to failing with timeout error. When Horizon is waiting for Keystone response, a I see in Keystone logs (/var/log/keystone/main.log) a lot of debug lines like this: `2015-09-17 11:37:47.990 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:47.991 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1067 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510` After that I've deleted almost all users from LDAP (there left only auto_user0, auto_user1 and auto_user10 and admin users) and tried to fetch them again with Horizon. And I've got the same lines: 2015-09-17 11:37:47.969 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510 2015-09-17 11:37:47.978 12575 DEBUG keystone.identity.core [-] Found existing mapping to public ID: a06df89598f65718695a3c3e138451f351416dfbd3f39fb254462f438de4c274 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:523 2015-09-17 11:37:47.980 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:47.982 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-10 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510 2015-09-17 11:37:47.989 12575 DEBUG keystone.identity.core [-] Found existing mapping to public ID: 44c396a392a341539101a28cdad4312d5113ca88c264fb4135550c0e336ae579 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:523 2015-09-17 11:37:47.990 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:47.991 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1067 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510 2015-09-17 11:37:47.998 12575 DEBUG keystone.identity.core [-] Found existing mapping to public ID: 85ca3743b8406ffd60505fa16342f66cf3f6b7ed9d8d197e3beab28e2715930c _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:523 2015-09-17 11:37:48.000 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:48.001 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1068 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510 2015-09-17 11:37:48.009 12575 DEBUG keystone.identity.core [-] Found existing mapping to public ID: 1de806c18b4cca49aefb0820e1097e8df5fd68ea0b1425b74643eacbc2e9d981 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:523 2015-09-17 11:37:48.011 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:48.012 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1069 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510 2015-09-17 11:37:48.019 12575 DEBUG keystone.identity.core [-] Found existing mapping to public ID: 94f6f8fbac83e4ae87fab63758322dabfaf43b9187a77f3b50d2bb66f26be5e4 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:523 2015-09-17 11:37:48.020 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:48.021 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1070 The problem is users auto_user-1070 , auto_user-1069, auto_user-1067, etc are no more present in LDAP. And even after 24 hours after users has been deleted from LDAP there are a lot of lines where Keystone says that it maps these users. And looks like while Keystone maps users, Horizon is waiting fore response. So, even if LDAP has not so much users, Horizon can get response from Keystone for listing users and fails with timeout. Here is how LDAP is configured with Keystone: /etc/keystone/domains/keystone.keystone.tld.conf contains the followong lines [ldap] user_allow_update=False user=cn=admin_ad,cn=Users,dc=keystone,dc=tld user_filter= user_name_attribute=cn user_pass_attribute=userPassword user_enabled_attribute=enabled suffix=dc=keystone,dc=tld password=Pass1234 url=ldap://172.16.57.78 user_allow_create=False user_allow_delete=False user_objectclass=person user_tree_dn=dc=keystone,dc=tld query_scope=sub user_id_attribute=cn debug_level=-1 page_size = 50 [identity] driver=keystone.identity.backends.ldap.Identity
2015-09-17 12:47:06 Vasyl Saienko mos: status New Confirmed
2015-09-17 12:47:20 Vasyl Saienko nominated for series mos/7.0.x
2015-09-17 12:47:20 Vasyl Saienko bug task added mos/7.0.x
2015-09-17 12:47:20 Vasyl Saienko nominated for series mos/8.0.x
2015-09-17 12:47:20 Vasyl Saienko bug task added mos/8.0.x
2015-09-17 12:47:30 Vasyl Saienko mos/7.0.x: status New Confirmed
2015-09-17 12:47:53 Vasyl Saienko mos/7.0.x: assignee MOS Keystone (mos-keystone)
2015-09-17 12:48:01 Vasyl Saienko mos/8.0.x: assignee MOS Keystone (mos-keystone)
2015-09-17 12:48:04 Vasyl Saienko mos/7.0.x: milestone 7.0
2015-09-17 12:48:06 Vasyl Saienko mos/8.0.x: milestone 8.0
2015-09-17 13:17:01 Vitaly Sedelnik mos/8.0.x: importance Undecided High
2015-09-17 13:17:03 Vitaly Sedelnik mos/7.0.x: importance Undecided High
2015-09-17 13:17:07 Vitaly Sedelnik mos/7.0.x: milestone 7.0 7.0-updates
2015-09-17 13:46:58 Paul Karikh description I've deployed MOS 7.0 with Fuel Keystone LDAP plugin and tried to list users with Horizon. Everything works slow, but fine and there were no errors. After that I've created 5K users in the LDAP. Somewhere after 1.5K users Horizon requests to Keystone started to failing with timeout error. When Horizon is waiting for Keystone response, a I see in Keystone logs (/var/log/keystone/main.log) a lot of debug lines like this: `2015-09-17 11:37:47.990 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:47.991 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1067 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510` After that I've deleted almost all users from LDAP (there left only auto_user0, auto_user1 and auto_user10 and admin users) and tried to fetch them again with Horizon. And I've got the same lines: 2015-09-17 11:37:47.969 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510 2015-09-17 11:37:47.978 12575 DEBUG keystone.identity.core [-] Found existing mapping to public ID: a06df89598f65718695a3c3e138451f351416dfbd3f39fb254462f438de4c274 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:523 2015-09-17 11:37:47.980 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:47.982 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-10 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510 2015-09-17 11:37:47.989 12575 DEBUG keystone.identity.core [-] Found existing mapping to public ID: 44c396a392a341539101a28cdad4312d5113ca88c264fb4135550c0e336ae579 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:523 2015-09-17 11:37:47.990 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:47.991 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1067 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510 2015-09-17 11:37:47.998 12575 DEBUG keystone.identity.core [-] Found existing mapping to public ID: 85ca3743b8406ffd60505fa16342f66cf3f6b7ed9d8d197e3beab28e2715930c _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:523 2015-09-17 11:37:48.000 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:48.001 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1068 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510 2015-09-17 11:37:48.009 12575 DEBUG keystone.identity.core [-] Found existing mapping to public ID: 1de806c18b4cca49aefb0820e1097e8df5fd68ea0b1425b74643eacbc2e9d981 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:523 2015-09-17 11:37:48.011 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:48.012 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1069 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510 2015-09-17 11:37:48.019 12575 DEBUG keystone.identity.core [-] Found existing mapping to public ID: 94f6f8fbac83e4ae87fab63758322dabfaf43b9187a77f3b50d2bb66f26be5e4 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:523 2015-09-17 11:37:48.020 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:48.021 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1070 The problem is users auto_user-1070 , auto_user-1069, auto_user-1067, etc are no more present in LDAP. And even after 24 hours after users has been deleted from LDAP there are a lot of lines where Keystone says that it maps these users. And looks like while Keystone maps users, Horizon is waiting fore response. So, even if LDAP has not so much users, Horizon can get response from Keystone for listing users and fails with timeout. Here is how LDAP is configured with Keystone: /etc/keystone/domains/keystone.keystone.tld.conf contains the followong lines [ldap] user_allow_update=False user=cn=admin_ad,cn=Users,dc=keystone,dc=tld user_filter= user_name_attribute=cn user_pass_attribute=userPassword user_enabled_attribute=enabled suffix=dc=keystone,dc=tld password=Pass1234 url=ldap://172.16.57.78 user_allow_create=False user_allow_delete=False user_objectclass=person user_tree_dn=dc=keystone,dc=tld query_scope=sub user_id_attribute=cn debug_level=-1 page_size = 50 [identity] driver=keystone.identity.backends.ldap.Identity I've deployed MOS 7.0 with Fuel Keystone LDAP plugin and tried to list users with Horizon. Everything works slow, but fine and there were no errors. After that I've created 5K users in the LDAP. Somewhere after 1.5K users Horizon requests to Keystone started to failing with timeout error. When Horizon is waiting for Keystone response, a I see in Keystone logs (/var/log/keystone/main.log) a lot of debug lines like this: `2015-09-17 11:37:47.990 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:47.991 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1067 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510` After that I've deleted almost all users from LDAP (there left only auto_user0, auto_user1 and auto_user10 and admin users) and tried to fetch them again with Horizon. And I've got the same lines: 2015-09-17 11:37:47.969 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510 2015-09-17 11:37:47.978 12575 DEBUG keystone.identity.core [-] Found existing mapping to public ID: a06df89598f65718695a3c3e138451f351416dfbd3f39fb254462f438de4c274 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:523 2015-09-17 11:37:47.980 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:47.982 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-10 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510 2015-09-17 11:37:47.989 12575 DEBUG keystone.identity.core [-] Found existing mapping to public ID: 44c396a392a341539101a28cdad4312d5113ca88c264fb4135550c0e336ae579 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:523 2015-09-17 11:37:47.990 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:47.991 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1067 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510 2015-09-17 11:37:47.998 12575 DEBUG keystone.identity.core [-] Found existing mapping to public ID: 85ca3743b8406ffd60505fa16342f66cf3f6b7ed9d8d197e3beab28e2715930c _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:523 2015-09-17 11:37:48.000 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:48.001 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1068 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510 2015-09-17 11:37:48.009 12575 DEBUG keystone.identity.core [-] Found existing mapping to public ID: 1de806c18b4cca49aefb0820e1097e8df5fd68ea0b1425b74643eacbc2e9d981 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:523 2015-09-17 11:37:48.011 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:48.012 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1069 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:510 2015-09-17 11:37:48.019 12575 DEBUG keystone.identity.core [-] Found existing mapping to public ID: 94f6f8fbac83e4ae87fab63758322dabfaf43b9187a77f3b50d2bb66f26be5e4 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:523 2015-09-17 11:37:48.020 12575 DEBUG keystone.identity.core [-] ID Mapping - Domain ID: cdb5ef2ad5324ba3b4a56cec6ac33420, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:492 2015-09-17 11:37:48.021 12575 DEBUG keystone.identity.core [-] Local ID: auto_user-1070 The problem is users auto_user-1070 , auto_user-1069, auto_user-1067, etc are no more present in LDAP. And even after 24 hours after users has been deleted from LDAP there are a lot of lines where Keystone says that it maps these users. And looks like while Keystone maps users, Horizon is waiting fore response. So, even if LDAP has not so much users, Horizon can't get a response from Keystone for listing users and fails with timeout. Here is how LDAP is configured with Keystone: /etc/keystone/domains/keystone.keystone.tld.conf contains the followong lines [ldap] user_allow_update=False user=cn=admin_ad,cn=Users,dc=keystone,dc=tld user_filter= user_name_attribute=cn user_pass_attribute=userPassword user_enabled_attribute=enabled suffix=dc=keystone,dc=tld password=Pass1234 url=ldap://172.16.57.78 user_allow_create=False user_allow_delete=False user_objectclass=person user_tree_dn=dc=keystone,dc=tld query_scope=sub user_id_attribute=cn debug_level=-1 page_size = 50 [identity] driver=keystone.identity.backends.ldap.Identity
2015-09-17 15:58:30 Timur Sufiev summary Keystone loggs that it maps already non-existent users Keystone logs that it maps already non-existent users
2015-09-25 16:38:16 Paul Karikh summary Keystone logs that it maps already non-existent users Keystone reads users from LDAP too slowly
2015-11-13 11:14:34 Alexander Makarov summary Keystone reads users from LDAP too slowly Keystone reads users from LDAP too slow
2015-11-13 11:15:01 Alexander Makarov mos/7.0.x: status Confirmed Triaged
2015-11-13 11:15:03 Alexander Makarov mos/8.0.x: status Confirmed Triaged
2015-11-26 09:19:18 Boris Bobrov mos/7.0.x: assignee MOS Keystone (mos-keystone) Boris Bobrov (bbobrov)
2015-11-26 09:19:21 Boris Bobrov mos/8.0.x: assignee MOS Keystone (mos-keystone) Boris Bobrov (bbobrov)
2015-11-26 10:23:20 Michael Kraynov tags customer-found
2015-11-26 10:31:48 Alex bug added subscriber Alex
2015-12-07 07:23:40 Boris Bobrov mos/8.0.x: status Triaged In Progress
2016-01-26 18:04:39 Dmytro Fomenko bug added subscriber Dmytro Fomenko
2016-02-02 10:13:33 Roman Podoliaka tags customer-found area-keystone customer-found move-to-9.0
2016-02-02 15:57:50 Roman Podoliaka tags area-keystone customer-found move-to-9.0 area-keystone customer-found enhancement move-to-9.0
2016-02-02 16:30:00 Boris Bobrov nominated for series mos/9.0.x
2016-02-02 16:30:00 Boris Bobrov bug task added mos/9.0.x
2016-02-02 16:30:07 Boris Bobrov mos/9.0.x: milestone 9.0
2016-02-02 16:30:13 Boris Bobrov mos/9.0.x: assignee Boris Bobrov (bbobrov)
2016-02-02 16:38:12 Roman Podoliaka tags area-keystone customer-found enhancement move-to-9.0 area-keystone customer-found enhancement move-to-9.0 release-notes
2016-02-02 16:41:37 Boris Bobrov mos/9.0.x: importance Undecided Medium
2016-02-02 16:41:39 Boris Bobrov mos/9.0.x: status New Confirmed
2016-02-02 16:42:05 Boris Bobrov mos/8.0.x: status In Progress Won't Fix
2016-02-02 16:42:08 Boris Bobrov mos/7.0.x: status Triaged Won't Fix
2016-02-03 18:08:33 Roman Podoliaka mos: status In Progress Won't Fix
2016-02-03 18:08:47 Roman Podoliaka mos/9.0.x: importance Medium High
2016-02-12 16:56:15 Vitaly Sedelnik tags area-keystone customer-found enhancement move-to-9.0 release-notes area-keystone customer-found enhancement move-to-9.0 release-notes wontfix-feature
2016-02-25 10:18:17 Olga Gusarenko tags area-keystone customer-found enhancement move-to-9.0 release-notes wontfix-feature 8.0 area-keystone customer-found enhancement move-to-9.0 release-notes-done wontfix-feature
2016-04-19 09:13:34 Fuel Devops McRobotson mos/10.0.x: importance Undecided High
2016-04-19 09:13:34 Fuel Devops McRobotson mos/10.0.x: status New Confirmed
2016-04-19 09:13:34 Fuel Devops McRobotson mos/10.0.x: milestone 10.0
2016-04-19 09:13:34 Fuel Devops McRobotson mos/10.0.x: assignee Boris Bobrov (bbobrov)
2016-05-31 10:40:31 Boris Bobrov mos/9.0.x: status Confirmed Won't Fix
2016-07-13 13:31:15 Boris Bobrov mos/9.x: status Won't Fix Confirmed
2016-07-13 13:31:22 Boris Bobrov mos/9.x: milestone 9.0 9.1
2016-07-13 13:31:37 Boris Bobrov tags 8.0 area-keystone customer-found enhancement move-to-9.0 release-notes-done wontfix-feature 10.0-reviewed 8.0 area-keystone customer-found enhancement move-to-9.0 release-notes-done wontfix-feature
2016-10-13 12:15:02 Roman Vyalov mos/9.x: milestone 9.1 9.2
2016-12-15 09:48:45 Maria Zlatkova tags 10.0-reviewed 8.0 area-keystone customer-found enhancement move-to-9.0 release-notes-done wontfix-feature 10.0-reviewed 8.0 area-keystone customer-found enhancement move-to-9.0 release-notes wontfix-feature
2017-01-26 10:37:32 Maria Zlatkova tags 10.0-reviewed 8.0 area-keystone customer-found enhancement move-to-9.0 release-notes wontfix-feature 10.0-reviewed 8.0 area-keystone customer-found enhancement move-to-9.0 release-notes-done wontfix-feature
2017-02-07 14:11:19 Vitaly Sedelnik mos/9.x: milestone 9.2 9.x-updates
2017-03-02 11:04:49 Denis Meltsaykin mos/9.x: status Confirmed Won't Fix
2022-05-10 20:06:53 Boris Bobrov mos/10.0.x: assignee Boris Bobrov (bbobrov)