ERROR oslo_messaging.rpc.dispatcher TrusteeCreateFailed: Failed to create trustee 720d0137-3dc6-4c12-a323-bcfc924466a1 in domain 61a6f2b431c8475dbcc40e756cf3d8d6

[narasimha18sv]

Unable to create bay for swarn bay model. I have enabled heat, lbaas and made sure the certificates to be taken care by local
magnum.conf:
[certificates]
cert_manager_type = local

[DEFAULT]
rpc_backend = rabbit
notification_driver = messaging
log_dir=/var/log/magnum

[api]

host = <IP>

[database]
connection=mysql://magnum:cloud123@<IP>/magnum

[keystone_authtoken]
auth_uri=http://<IP>:5000/v2.0
identity_uri=http://<IP>:35357
auth_strategy=keystone
admin_user=magnum
admin_password=cloud123
admin_tenant_name=service

[oslo_messaging_rabbit]
rabbit_host = <IP>
rabbit_userid = openstack
rabbit_password = cloud123
rabbit_virtual_host = /

I have installed this manually by using ubuntu mitaka repository.

Below is the stack trace of the error in mangum-conductor log

2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone [req-6ce9a0e2-f0d7-479f-a9b1-3cea3f020ce6 admin admin - - -] Failed to create trustee
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone Traceback (most recent call last):
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File "/usr/lib/python2.7/dist-packages/magnum/common/keystone.py", line 184, in create_trustee
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone domain=domain_id)
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File "/usr/lib/python2.7/dist-packages/debtcollector/renames.py", line 45, in wrapper
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone return f(*args, **kwargs)
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone return func(*args, **kwargs)
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/users.py", line 77, in create
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone log=not bool(password))
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 171, in _post
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone resp, body = self.client.post(url, body=body, **kwargs)
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 179, in post
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone return self.request(url, 'POST', **kwargs)
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 331, in request
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone return self.session.request(url, method, **kwargs)
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone return func(*args, **kwargs)
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 320, in request
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone auth_headers = self.get_auth_headers(auth)
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 638, in get_auth_headers
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone return auth.get_headers(self, **kwargs)
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/base.py", line 159, in get_headers
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone token = self.get_token(session)
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 216, in get_token
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone return self.get_access(session).auth_token
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 262, in get_access
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone self.auth_ref = self.get_auth_ref(session)
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/v3/base.py", line 190, in get_auth_ref
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone authenticated=False, log=False, **rkwargs)
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 520, in post
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone return self.request(url, 'POST', **kwargs)
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone return func(*args, **kwargs)
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 420, in request
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone raise exceptions.from_response(resp, method, url)
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone NotFound: The resource could not be found. (HTTP 404)
2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone
2016-04-26 12:10:46.062 19391 ERROR oslo_messaging.rpc.dispatcher [req-6ce9a0e2-f0d7-479f-a9b1-3cea3f020ce6 admin admin - - -] Exception during message handling: Failed to create trustee 720d0137-3dc6-4c12-a323-bcfc924466a1 in domain 61a6f2b431c8475dbcc40e756cf3d8d6
2016-04-26 12:10:46.062 19391 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-04-26 12:10:46.062 19391 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-04-26 12:10:46.062 19391 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2016-04-26 12:10:46.062 19391 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-04-26 12:10:46.062 19391 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-04-26 12:10:46.062 19391 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-04-26 12:10:46.062 19391 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-04-26 12:10:46.062 19391 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/magnum/conductor/handlers/bay_conductor.py", line 143, in bay_create
2016-04-26 12:10:46.062 19391 ERROR oslo_messaging.rpc.dispatcher self._create_trustee_and_trust(osc, bay)
2016-04-26 12:10:46.062 19391 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/magnum/conductor/handlers/bay_conductor.py", line 128, in _create_trustee_and_trust
2016-04-26 12:10:46.062 19391 ERROR oslo_messaging.rpc.dispatcher CONF.trust.trustee_domain_id)
2016-04-26 12:10:46.062 19391 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/magnum/common/keystone.py", line 188, in create_trustee
2016-04-26 12:10:46.062 19391 ERROR oslo_messaging.rpc.dispatcher domain_id=domain_id)
2016-04-26 12:10:46.062 19391 ERROR oslo_messaging.rpc.dispatcher TrusteeCreateFailed: Failed to create trustee 720d0137-3dc6-4c12-a323-bcfc924466a1 in domain 61a6f2b431c8475dbcc40e756cf3d8d6
2016-04-26 12:10:46.062 19391 ERROR oslo_messaging.rpc.dispatcher
2016-04-26 12:10:46.071 19391 ERROR oslo_messaging._drivers.common [req-6ce9a0e2-f0d7-479f-a9b1-3cea3f020ce6 admin admin - - -] Returning exception Failed to create trustee 720d0137-3dc6-4c12-a323-bcfc924466a1 in domain 61a6f2b431c8475dbcc40e756cf3d8d6 to caller

[Michal Adamczyk (vanditboy)]

Hi,

I have the same error on CentOS 7 and Mitaka release (fresh install on 05.05.2016 from official Centos repos):

controller:root:~> magnum bay-create --name swarmbay --baymodel swarmbaymodel --node-count 2
Create for bay swarmbay failed: Internal Server Error (HTTP 500) (Request-ID: req-08d3dc32-cb70-4b72-9c13-1ffba08734b8)

In the logs I see:

2016-05-06 09:27:40.374 9803 ERROR wsme.api [req-f46cf3d5-a51c-4963-bad9-b541cee0db9d admin openstack - - -] Server-side error: "Failed to create trustee 811b2d2b-cd37-4edc-895f-a23d8b91d092 in domain a1059213d5b14ac097aa70ef2bbabcb2
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
    incoming.message))

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
    result = func(ctxt, **new_args)

  File "/usr/lib/python2.7/site-packages/magnum/conductor/handlers/bay_conductor.py", line 143, in bay_create
    self._create_trustee_and_trust(osc, bay)

  File "/usr/lib/python2.7/site-packages/magnum/conductor/handlers/bay_conductor.py", line 128, in _create_trustee_and_trust
    CONF.trust.trustee_domain_id)

  File "/usr/lib/python2.7/site-packages/magnum/common/keystone.py", line 188, in create_trustee
    domain_id=domain_id)

TrusteeCreateFailed: Failed to create trustee 811b2d2b-cd37-4edc-895f-a23d8b91d092 in domain a1059213d5b14ac097aa70ef2bbabcb2
". Detail:
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/wsmeext/pecan.py", line 84, in callfunction
    result = f(self, *args, **kwargs)

  File "/usr/lib/python2.7/site-packages/magnum/api/controllers/v1/bay.py", line 299, in post
    bay.bay_create_timeout)

  File "/usr/lib/python2.7/site-packages/magnum/conductor/api.py", line 35, in bay_create
    bay_create_timeout=bay_create_timeout)

  File "/usr/lib/python2.7/site-packages/magnum/common/rpc_service.py", line 110, in _call
    return self._client.call(self._context, method, *args, **kwargs)

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 413, in call
    return self.prepare().call(ctxt, method, **kwargs)

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
    retry=self.retry)

  File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
    timeout=timeout, retry=retry)

  File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
    retry=retry)

  File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 461, in _send
    raise result

TrusteeCreateFailed_Remote: Failed to create trustee 811b2d2b-cd37-4edc-895f-a23d8b91d092 in domain a1059213d5b14ac097aa70ef2bbabcb2
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
    incoming.message))

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py...

[Michal Adamczyk (vanditboy)]

rpm list from the system

[Michal Adamczyk (vanditboy)]

magnum.conf

[hongbin (hongbin034)]

@Michal,

I saw you have this line in your config file:

admin_tenant_name=services

How about changing it to:

admin_tenant_name=services (no 's')

[hongbin (hongbin034)]

Sorry, I mean change it to (no 's'):

admin_tenant_name=service

[hongbin (hongbin034)]

For your reference, I attached my config file

[Michal Adamczyk (vanditboy)]

Hi hongbin,

Thanks for the hing but I am using "services" project inside my OpenStack installation.

I have the same in nova, glance.

Kind regards,
Michal.

[Michal Adamczyk (vanditboy)]

Hi,

Can anyone has this issue?

Kind regards,
Michal.

[hongbin (hongbin034)]

@Michal,

In your Magnum config file, I saw you set the following:

  [keystone_authtoken]
  auth_uri = http://172.19.0.10:5000

Could you append the '/v3' to the URL:

  [keystone_authtoken]
  auth_uri = http://172.19.0.10:5000/v3

Then restart magnum-api and magnum-cond and try again. Let me know if it still fail.

[Michal Adamczyk (vanditboy)]

Hi,

I have:

auth_version = v3

When I did change the URL I was getting error and nothing works:

2016-05-10 22:59:48.445 25359 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2016-05-10 22:59:48.453 25359 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}}
2016-05-10 22:59:48.454 25359 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2016-05-10 22:59:48.617 25359 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2016-05-10 22:59:48.618 25359 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}}
2016-05-10 22:59:48.618 25359 WARNING keystonemiddleware.auth_token [-] Authorization failed for token

[hongbin (hongbin034)]

How about adding the following configs (replace the values with yours):

[keystone_auth]
auth_url = http://192.168.10.100:5000/v3
user_domain_id = default
project_domain_id = default
project_name = service
password = password
username = magnum
auth_type = password

[Michal Adamczyk (vanditboy)]

@hongbin,

In fact your suggestion from https://bugs.launchpad.net/magnum/+bug/1575524/comments/9 works till some point, bellow my config:

http://paste.openstack.org/show/496652/

Now I am getting following error:

magnum bay-create --name swarmbay --baymodel swarmbaymodel --node-count 2

and the magnum-conductor.log:

2016-05-11 10:46:49.038 10008 ERROR oslo_messaging.rpc.dispatcher
2016-05-11 10:46:49.039 10008 ERROR oslo_messaging._drivers.common [req-cadfe8f1-7479-4024-8774-92dc7162ad3d admin openstack - - -] Returning exception ERROR: HEAT-E99001 Service neutron is not available for resource type OS::Neutron::HealthMonitor, reason: Service endpoint not in service catalog. to caller
2016-05-11 10:46:49.040 10008 ERROR oslo_messaging._drivers.common [req-cadfe8f1-7479-4024-8774-92dc7162ad3d admin openstack - - -] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python2.7/site-packages/magnum/conductor/handlers/bay_conductor.py", line 151, in bay_create\n raise exception.InvalidParameterValue(message=six.text_type(e))\n', 'InvalidParameterValue: ERROR: HEAT-E99001 Service neutron is not available for resource type OS::Neutron::HealthMonitor, reason: Service endpoint not in service catalog.\n']

I have switched to newton branch with a clean installation and the same story...

[narasimha18sv (narasimha18sv)]

@Michal
The above error is due to neutron-lbaas-agent
Install and configure it . Magnum requires lbaas service

[narasimha18sv (narasimha18sv)]

How did you resolve above error of trust. Please post the resolution here as I am.getting the same error with ubuntu repo mitaka release

[Michal Adamczyk (vanditboy)]

Hi,

Working config is inside: http://paste.openstack.org/show/496652/

Kind regards,
Michal.

[Michal Adamczyk (vanditboy)]

Hi Everyone,

I did enabled lbaasv2 for Mitaka as apparently this is the only supported version according to the docs: http://docs.openstack.org/mitaka/networking-guide/adv-config-lbaas.html

Surprise, surprise - Magnum supports only lbaasv1! More info in side this thread: http://lists.openstack.org/pipermail/openstack-dev/2016-May/094714.html

I am still with following error:

ERROR: HEAT-E99001 Service neutron is not available for resource type OS::Neutron::HealthMonitor, reason: Service endpoint not in service catalog.

Neutron agent-list shows that the lb agent is enabled

[Spyros Trigazis (strigazi)]

@Michal
Have you solved this issue?
AFAIK, trust users are not fully supported in mitaka.

[narasimha18sv (narasimha18sv)]

Sorry I am not working on it . I didn't check after that

On 16-Jul-2016 9:15 PM, "Spyros Trigazis" <email address hidden> wrote:

> @Michal
> Have you solved this issue?
> AFAIK, trust users are not fully supported in mitaka.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1575524
>
> Title:
> ERROR oslo_messaging.rpc.dispatcher TrusteeCreateFailed: Failed to
> create trustee 720d0137-3dc6-4c12-a323-bcfc924466a1 in domain
> 61a6f2b431c8475dbcc40e756cf3d8d6
>
> Status in Magnum:
> New
>
> Bug description:
> Unable to create bay for swarn bay model. I have enabled heat, lbaas and
> made sure the certificates to be taken care by local
> magnum.conf:
> [certificates]
> cert_manager_type = local
>
> [DEFAULT]
> rpc_backend = rabbit
> notification_driver = messaging
> log_dir=/var/log/magnum
>
> [api]
>
> host = <IP>
>
> [database]
> connection=mysql://magnum:cloud123@<IP>/magnum
>
>
> [keystone_authtoken]
> auth_uri=http://<IP>:5000/v2.0
> identity_uri=http://<IP>:35357
> auth_strategy=keystone
> admin_user=magnum
> admin_password=cloud123
> admin_tenant_name=service
>
>
> [oslo_messaging_rabbit]
> rabbit_host = <IP>
> rabbit_userid = openstack
> rabbit_password = cloud123
> rabbit_virtual_host = /
>
> I have installed this manually by using ubuntu mitaka repository.
>
> Below is the stack trace of the error in mangum-conductor log
>
> 2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone
> [req-6ce9a0e2-f0d7-479f-a9b1-3cea3f020ce6 admin admin - - -] Failed to
> create trustee
> 2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone Traceback
> (most recent call last):
> 2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File
> "/usr/lib/python2.7/dist-packages/magnum/common/keystone.py", line 184, in
> create_trustee
> 2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone
> domain=domain_id)
> 2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File
> "/usr/lib/python2.7/dist-packages/debtcollector/renames.py", line 45, in
> wrapper
> 2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone return
> f(*args, **kwargs)
> 2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File
> "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
> 2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone return
> func(*args, **kwargs)
> 2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File
> "/usr/lib/python2.7/dist-packages/keystoneclient/v3/users.py", line 77, in
> create
> 2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone log=not
> bool(password))
> 2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File
> "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 171, in
> _post
> 2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone resp,
> body = self.client.post(url, body=body, **kwargs)
> 2016-04-26 12:10:46.056 19391 ERROR magnum.common.keystone File
> "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 179, in
> post
> 2016-04-26 12:10:46.056 19391 ...

[hongbin (hongbin034)]

It looks this issue is resolved? I am going to close this bug now, but feel free to reopen it if the issue still occurs.