Keystone authentication dies during introspection or deploy on baremetal hardware

[Ronelle Landy]

With the latest Newton code installed, authentication errors show up during introspection or deployment, after which, no action requiring authentication can be processed:

[stack@undercloud ~]$ ironic node-list
Must provide Keystone credentials or user-defined endpoint and token, error was: Unable to establish connection to http://<ip>:5000/v2.0/tokens

[stack@undercloud ~]$ source stackrc
[stack@undercloud ~]$ openstack service list
Discovering versions from the identity service failed when creating the password plugin. Attempting to determine version from URL.
Unable to establish connection to http://<ip>:5000/v2.0/tokens

The fewer overcloud nodes that are used, the less likely you are to see the errors. When more than four overcloud nodes are used with multiple cpus each, error almost always appear at some point during introspection or deploy.

Restarting service that runs Keystone (apache) does not help.

In journalctl, there are errors looged from Keystone v2 and v3:

2016-06-20 20:50:09.092 10619 DEBUG keystone.middleware.auth [req-cc510e30-1aa2-47c1-bb88-7f5395e411f5 - - - - -] Authenticating user token process_request /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:509
2016-06-20 20:50:07.746 10619 INFO keystone.common.wsgi [req-6503f8f3-1a95-41c1-b079-1856c9bc9d15 269f112ab6e54b02b908938f4b3a2d09 4047ad48f0bc42789b334e56591cbe61 - default default] GET http://<ip>:35357/v3/auth/tokens

2016-06-20 20:50:07.565 10619 INFO keystone.common.wsgi [req-484e6ec7-4546-409c-b04e-104ac511fb2d - - - - -] GET http://<ip>:35357/
2016-06-20 20:50:07.574 10619 DEBUG keystone.middleware.auth [req-cc510e30-1aa2-47c1-bb88-7f5395e411f5 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. fill_context /usr/lib/python2.7/site-packages/keystone/middleware/auth.py:199
2016-06-20 20:50:07.575 10619 INFO keystone.common.wsgi [req-cc510e30-1aa2-47c1-bb88-7f5395e411f5 - - - - -] POST http://<ip>:35357/v2.0/tokens

----------------------------------

versions installed:

openstack-nova-cert-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-mistral-common-3.0.0-0.20160609153930.88b3994.el7.centos.noarch
puppet-openstacklib-8.0-0.20160604215800.6a89264.el7.centos.noarch
openstack-nova-compute-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-gnocchi-carbonara-2.1.1-0.20160608165716.0bbf079.el7.centos.noarch
openstack-aodh-common-2.0.1-0.20160607164941.77238e3.el7.centos.noarch
openstack-manila-3.0.0-0.20160608190848.3dfc4dc.el7.centos.noarch
openstack-swift-proxy-2.7.1-0.20160608221859.11c5ef7.el7.centos.noarch
openstack-nova-api-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-swift-container-2.7.1-0.20160608221859.11c5ef7.el7.centos.noarch
python2-openstacksdk-0.8.7-0.20160609094719.a93ceb2.el7.centos.noarch
openstack-puppet-modules-8.0.0-0.20160520142355.6a3e8bf.el7.centos.noarch
openstack-gnocchi-api-2.1.1-0.20160608165716.0bbf079.el7.centos.noarch
openstack-neutron-metering-agent-9.0.0-0.20160609100034.4272cfc.el7.centos.noarch
openstack-aodh-evaluator-2.0.1-0.20160607164941.77238e3.el7.centos.noarch
openstack-heat-api-7.0.0-0.20160609065225.1e47fac.el7.centos.noarch
openstack-glance-13.0.0-0.20160609161924.551018a.el7.centos.noarch
openstack-swift-2.7.1-0.20160608221859.11c5ef7.el7.centos.noarch
openstack-tripleo-0.0.1-0.20160427104752.b076a5a.el7.centos.noarch
openstack-mistral-api-3.0.0-0.20160609153930.88b3994.el7.centos.noarch
openstack-nova-common-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-nova-conductor-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-nova-cells-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-sahara-api-5.0.0-0.20160609071340.165172f.el7.centos.noarch
openstack-neutron-9.0.0-0.20160609100034.4272cfc.el7.centos.noarch
openstack-gnocchi-common-2.1.1-0.20160608165716.0bbf079.el7.centos.noarch
openstack-tempest-12.0.1-0.20160608094412.ebe375c.el7.centos.noarch
python2-django-openstack-auth-2.3.1-0.20160609091907.d2b9c5a.el7.centos.noarch
openstack-neutron-openvswitch-9.0.0-0.20160609100034.4272cfc.el7.centos.noarch
openstack-aodh-expirer-2.0.1-0.20160607164941.77238e3.el7.centos.noarch
openstack-ironic-conductor-5.1.1-0.20160609093258.d58e2bc.el7.centos.noarch
openstack-heat-engine-7.0.0-0.20160609065225.1e47fac.el7.centos.noarch
openstack-ironic-inspector-3.3.1-0.20160607104948.457c127.el7.centos.noarch
openstack-ceilometer-api-6.0.1-0.20160608200830.726f243.el7.centos.noarch
openstack-ceilometer-common-6.0.1-0.20160608200830.726f243.el7.centos.noarch
openstack-swift-account-2.7.1-0.20160608221859.11c5ef7.el7.centos.noarch
openstack-keystone-10.0.0-0.20160609012819.eafa8b6.el7.centos.noarch
openstack-selinux-0.7.2-1.el7.noarch
openstack-mistral-executor-3.0.0-0.20160609153930.88b3994.el7.centos.noarch
openstack-nova-scheduler-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-swift-plugin-swift3-1.10-1.el7.noarch
openstack-nova-novncproxy-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-sahara-engine-5.0.0-0.20160609071340.165172f.el7.centos.noarch
openstack-heat-common-7.0.0-0.20160609065225.1e47fac.el7.centos.noarch
openstack-ironic-common-5.1.1-0.20160609093258.d58e2bc.el7.centos.noarch
openstack-cinder-9.0.0-0.20160608231321.0d2e9cc.el7.centos.noarch
puppet-openstack_extras-8.0-0.20160603174417.f9aebca.el7.centos.noarch
openstack-nova-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-tripleo-common-5.0.0-0.20160609081839.fbc0ee3.el7.centos.noarch
openstack-dashboard-10.0.0-0.20160609090113.ce68bf6.el7.centos.noarch
openstack-gnocchi-metricd-2.1.1-0.20160608165716.0bbf079.el7.centos.noarch
openstack-neutron-lbaas-9.0.0-0.20160609054819.22f5290.el7.centos.noarch
openstack-aodh-listener-2.0.1-0.20160607164941.77238e3.el7.centos.noarch
openstack-ironic-api-5.1.1-0.20160609093258.d58e2bc.el7.centos.noarch
openstack-heat-api-cloudwatch-7.0.0-0.20160609065225.1e47fac.el7.centos.noarch
openstack-ceilometer-central-6.0.1-0.20160608200830.726f243.el7.centos.noarch
openstack-ceilometer-compute-6.0.1-0.20160608200830.726f243.el7.centos.noarch
openstack-ceilometer-polling-6.0.1-0.20160608200830.726f243.el7.centos.noarch
openstack-zaqar-3.0.0-0.20160608232004.cb0e04a.el7.centos.noarch
openstack-nova-console-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-sahara-common-5.0.0-0.20160609071340.165172f.el7.centos.noarch
python-openstackclient-2.6.1-0.20160609034343.db0849e.el7.centos.noarch
openstack-gnocchi-statsd-2.1.1-0.20160608165716.0bbf079.el7.centos.noarch
openstack-neutron-ml2-9.0.0-0.20160609100034.4272cfc.el7.centos.noarch
openstack-aodh-notifier-2.0.1-0.20160607164941.77238e3.el7.centos.noarch
openstack-heat-api-cfn-7.0.0-0.20160609065225.1e47fac.el7.centos.noarch
openstack-ceilometer-collector-6.0.1-0.20160608200830.726f243.el7.centos.noarch
openstack-ceilometer-notification-6.0.1-0.20160608200830.726f243.el7.centos.noarch
openstack-tripleo-puppet-elements-5.0.0-0.20160607234408.3b2ddd9.el7.centos.noarch
openstack-mistral-engine-3.0.0-0.20160609153930.88b3994.el7.centos.noarch
openstack-nova-network-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-sahara-5.0.0-0.20160609071340.165172f.el7.centos.noarch
python-openstack-mistral-3.0.0-0.20160609153930.88b3994.el7.centos.noarch
openstack-tripleo-image-elements-5.0.0-0.20160607234521.3053ef4.el7.centos.noarch
openstack-tripleo-heat-templates-5.0.0-0.20160609160634.8027ae2.el7.centos.noarch
openstack-heat-templates-0.0.1-0.20160608123220.c9c5e15.el7.centos.noarch
openstack-neutron-common-9.0.0-0.20160609100034.4272cfc.el7.centos.noarch
openstack-gnocchi-indexer-sqlalchemy-2.1.1-0.20160608165716.0bbf079.el7.centos.noarch
openstack-aodh-api-2.0.1-0.20160607164941.77238e3.el7.centos.noarch
openstack-manila-share-3.0.0-0.20160608190848.3dfc4dc.el7.centos.noarch
openstack-utils-2015.2-1.el7.noarch
openstack-aodh-compat-2.0.1-0.20160607164941.77238e3.el7.centos.noarch
openstack-swift-object-2.7.1-0.20160608221859.11c5ef7.el7.centos.noarch

It is possible that this is configuration issue and that if the config values are more suited to the environment, we could complete deploy. If so, this would be a doc request.

[Ronelle Landy (rlandy)]

Another trace:

[stack@undercloud ~]$ openstack baremetal introspection bulk start
Setting nodes for introspection to manageable...
Starting introspection of node: 5e7af372-42fc-4109-a6a5-de5bf8080014

Internal server error (ConnectionRefused): Error has occurred while handling request for http://<ip>:6385/v1/nodes/5e7af372-42fc-4109-a6a5-de5bf8080014: HTTPConnectionPool(host='<ip>', port=6385): Max retries exceeded with url: /v1/nodes/5e7af372-42fc-4109-a6a5-de5bf8080014 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x4456e10>: Failed to establish a new connection: [Errno 113] EHOSTUNREACH',))

[Ronelle Landy (rlandy)]

So it seems that these errors are related to a lack of resources on the undercloud VM.

Increasing the resources - vcpus, memory, disk space solves the problem.

This should be doc'ed for deploying on baremetal hardware with virt undercloud - the default size of the VM is not sufficient.

[Emilien Macchi (emilienm)]

This bug was last updated over 180 days ago, as tripleo is a fast moving project and we'd like to get the tracker down to currently actionable bugs, this is getting marked as Invalid. If the issue still exists, please feel free to reopen it.