[1.06-70] SI launch is failing as svc-mon is looking to authenticate user in a stale/non-existent tenant

[Ganesha HV]

1]. Created a project 'TestECMPSvcChain0' and added a user 'TestECMPSvcChain0' as a admin and also 'admin' as a Member.
2]. Created a service-instance with service-scaling and launched 3 instances in the same.
3]. The service-instance is not launched successfully with the following error :

Traceback (most recent call last):
  File "/opt/contrail/api-venv/lib/python2.7/site-packages/svc_monitor/svc_monitor.py", line 1015, in launch_arc
    monitor.process_poll_result(result)
  File "/opt/contrail/api-venv/lib/python2.7/site-packages/svc_monitor/svc_monitor.py", line 857, in process_poll_result
    func(idents)
  File "/opt/contrail/api-venv/lib/python2.7/site-packages/svc_monitor/svc_monitor.py", line 797, in _addmsg_service_instance_service_template
    self._create_svc_instance_vm(st_obj, si_obj)
  File "/opt/contrail/api-venv/lib/python2.7/site-packages/svc_monitor/svc_monitor.py", line 495, in _create_svc_instance_vm
    nics, proj_obj.name, flavor)
  File "/opt/contrail/api-venv/lib/python2.7/site-packages/svc_monitor/svc_monitor.py", line 905, in _create_svc_vm
    flavor = n_client.flavors.find(name=flavor_name)
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 184, in find
    matches = self.findall(**kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 222, in findall
    listing = self.list(**list_kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/flavors.py", line 103, in list
    return self._list("/flavors%s%s" % (detail, query_string), "flavors")
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 61, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 229, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 202, in _cs_request
    self.authenticate()
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 329, in authenticate
    auth_url = self._v2_auth(auth_url)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 411, in _v2_auth
    return self._authenticate(url, body)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 423, in _authenticate
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 195, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 189, in request
    raise exceptions.from_response(resp, body, url, method)
Unauthorized: User 5bb33e7bb42d49f9a68978ed505b6ddd is unauthorized for tenant 84e8958626634a97948fcfa22bdc4401 (HTTP 401)

4]. It is seen that though the user exists, the tenant it is referring to is non-existent :

(api-venv)root@nodec4:/var/log/contrail# keystone user-list
WARNING: Bypassing authentication using a token & endpoint (authentication credentials are being ignored).
+----------------------------------+-------------------+---------+---------------------+
| id | name | enabled | email |
+----------------------------------+-------------------+---------+---------------------+
| f33ae47ca65b4fe18820b539a3413526 | TestECMPSvcChain0 | True | |
| 5bb33e7bb42d49f9a68978ed505b6ddd | admin | True | <email address hidden> |
| 19b66aa3d2514dd78fdf401bae3642ce | cinder | True | <email address hidden> |
| c95afaf99d2f44b1996cb95f334c7024 | demo | True | <email address hidden> |
| 2e2b26c3122245ef86bf3f2115dd260f | glance | True | <email address hidden> |
| 99f152ece64e451c8ddb363331af65c6 | neutron | True | <email address hidden> |
| 5654ddf0744148998c2d9fafece00775 | nova | True | <email address hidden> |
+----------------------------------+-------------------+---------+---------------------+
(api-venv)root@nodec4:/var/log/contrail# keystone tenant-list
WARNING: Bypassing authentication using a token & endpoint (authentication credentials are being ignored).
+----------------------------------+--------------------+---------+
| id | name | enabled |
+----------------------------------+--------------------+---------+
| 979cb99dc0d74fdc81ded28765317dac | TestECMPSvcChain0 | True |
| e66fb526c06c4c9a98b110d9ac206936 | admin | True |
| dc4ce1a074b14f3e831b43ab863cb83a | demo | True |
| 3488872267bf4beab977519f97c64be4 | invisible_to_admin | True |
| 9cab150dec974174a194ad88e562043d | service | True |
+----------------------------------+--------------------+---------+

I have kept the logs at
/home/bhushana/Documents/technical/bugs/<bug-id> on 10.204.216.50

Login with the following credentials:

USN : bhushana
PWD : bhu@123

[Ganesha HV (ganeshahv)]

Steps to automate:
=================

- Create a Tenant in Keystone.
- Create a VN in this tenant in Contrail
- Create a Service Instance in this tenant, with a least one leg in a network from this Tenant.
- Delete the SI
- Delete the VN
- Delete the Tenant.

- recreate the tenant with the same name
- recreate the VN with the same name
- recreate the same Service Instance

=> the Service Instance remains in "Spawing" state.