"Cannot remove role that has not been granted" when try to delete _member_ role of heat user during deploy cluster

[Leontii Istomin]

Deploy cluster has been failed on the first controller node with the following error:
Execution of '/usr/bin/openstack role remove _member_ --project 5972e36d64d64459969dfb9078491366 --user dc0de59470e1483bae401eb0acc69271' returned 1: ERROR: openstack Could not find role assignment with role: 9fe2ff9ee4384b1894a90878d3e92bab, user or group: dc0de59470e1483bae401eb0acc69271, project or domain: 5972e36d64d64459969dfb9078491366 (HTTP 404) (Request-ID: req-f92965e8-9e7b-47f2-8b26-6c50624c031d)

tried to execute this command manually: http://paste.openstack.org/show/467539/
users, tenants, roles and roles of heat user: http://paste.openstack.org/show/467541/

_member_ role hasn't been assigned to user heat at the time

Cluster Configuration:
Baremetal, Ubuntu,IBP,HA,Neutron-vxlan,Ceph-all,Nova-debug,Nova-quotas,7.0-297
Controllers:3 Computes:178 Copmutes+Ceph: 20 LMA:2

api: '1.0'
astute_sha: 6c5b73f93e24cc781c809db9159927655ced5012
auth_required: true
build_id: '297'
build_number: '297'
feature_groups:
- mirantis
fuel-agent_sha: 082a47bf014002e515001be05f99040437281a2d
fuel-library_sha: 0623b4daad438ceeb5dc41b10cdd3011795fff7e
fuel-nailgun-agent_sha: d7027952870a35db8dc52f185bb1158cdd3d1ebd
fuel-ostf_sha: 1f08e6e71021179b9881a824d9c999957fcc7045
fuelmain_sha: 6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85
nailgun_sha: 28751b8cb0f2300cb3a036abc70f75c422f69fd7
openstack_version: 2015.1.0-7.0
production: docker
python-fuelclient_sha: 2864459e27b0510a0f7aedac6cdf27901ef5c481
release: '7.0'

Diagnostic Snapshot: http://mos-scale-share.mirantis.com/fuel-snapshot-2015-09-18_08-03-54.tar.xz

[Matthew Mosesohn (raytrac3r)]

Need snapshot

[Matthew Mosesohn (raytrac3r)]

Leontiy gave me access to the env. Node-93 deployed and reached the end of all tasks okay as far as I can tell. The one failure was this:
2015-09-17T21:34:37.411659+00:00 err: (/Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user_role[heat@services]/roles) change from ["_member_"] to admin failed: Execution of '/usr/bin/openstack role remove _member_ --project 5972e36d64d64459969dfb9078491366 --user dc0de59470e1483bae401eb0acc69271' returned 1: ERROR: openstack Could not find role assignment with role: 9fe2ff9ee4384b1894a90878d3e92bab, user or group: dc0de59470e1483bae401eb0acc69271, project or domain: 5972e36d64d64459969dfb9078491366 (HTTP 404) (Request-ID: req-f92965e8-9e7b-47f2-8b26-6c50624c031d)

but after a repeat of heat/keystone.pp task, it worked:

2015-09-17T21:29:55.255948+00:00 notice: (Scope(Class[main])) MODULAR: keystone.pp
2015-09-17T21:32:42.256139+00:00 notice: (Scope(Class[main])) MODULAR: heat/keystone.pp
2015-09-17T21:35:21.580823+00:00 notice: (Scope(Class[main])) MODULAR: heat/keystone.pp
2015-09-17T21:36:24.268274+00:00 notice: (Scope(Class[main])) MODULAR: cinder/keystone.pp

The astute.yaml is quite large, so still trying to figure out what else failed

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/225607

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matthew Mosesohn (<email address hidden>) on branch: master
Review: https://review.openstack.org/225607
Reason: Apparently this is now fixed in openstackclient with its own built in timeout

[Matthew Mosesohn (raytrac3r)]

This was fixed in https://review.fuel-infra.org/#/c/11896/

[Matthew Mosesohn (raytrac3r)]

https://review.fuel-infra.org/#/c/11896/ was merged before 7.0 GA

[Tatyanka (tatyana-leontovich)]

verified VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "466"
  build_id: "466"
  fuel-nailgun_sha: "f81311bbd6fee2665e3f96dcac55f72889b2f38c"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "6823f1d4005a634b8436109ab741a2194e2d32e0"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "fe03d887361eb80232e9914eae5b8d54304df781"
  fuel-ostf_sha: "ab5fd151fc6c1aa0b35bc2023631b1f4836ecd61"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "fac143f4dfa75785758e72afbdc029693e94ff2b"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "727f7076f04cb0caccc9f305b149a2b5b5c2af3a"