With the latest Newton code installed, authentication errors show up during introspection or deployment, after which, no action requiring authentication can be processed:
[stack@undercloud ~]$ ironic node-list
Must provide Keystone credentials or user-defined endpoint and token, error was: Unable to establish connection to http://<ip>:5000/v2.0/tokens
[stack@undercloud ~]$ source stackrc
[stack@undercloud ~]$ openstack service list
Discovering versions from the identity service failed when creating the password plugin. Attempting to determine version from URL.
Unable to establish connection to http://<ip>:5000/v2.0/tokens
The fewer overcloud nodes that are used, the less likely you are to see the errors. When more than four overcloud nodes are used with multiple cpus each, error almost always appear at some point during introspection or deploy.
Restarting service that runs Keystone (apache) does not help.
In journalctl, there are errors looged from Keystone v2 and v3:
2016-06-20 20:50:09.092 10619 DEBUG keystone.middleware.auth [req-cc510e30-1aa2-47c1-bb88-7f5395e411f5 - - - - -] Authenticating user token process_request /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:509
2016-06-20 20:50:07.746 10619 INFO keystone.common.wsgi [req-6503f8f3-1a95-41c1-b079-1856c9bc9d15 269f112ab6e54b02b908938f4b3a2d09 4047ad48f0bc42789b334e56591cbe61 - default default] GET http://<ip>:35357/v3/auth/tokens
2016-06-20 20:50:07.565 10619 INFO keystone.common.wsgi [req-484e6ec7-4546-409c-b04e-104ac511fb2d - - - - -] GET http://<ip>:35357/
2016-06-20 20:50:07.574 10619 DEBUG keystone.middleware.auth [req-cc510e30-1aa2-47c1-bb88-7f5395e411f5 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. fill_context /usr/lib/python2.7/site-packages/keystone/middleware/auth.py:199
2016-06-20 20:50:07.575 10619 INFO keystone.common.wsgi [req-cc510e30-1aa2-47c1-bb88-7f5395e411f5 - - - - -] POST http://<ip>:35357/v2.0/tokens
----------------------------------
versions installed:
openstack-nova-cert-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-mistral-common-3.0.0-0.20160609153930.88b3994.el7.centos.noarch
puppet-openstacklib-8.0-0.20160604215800.6a89264.el7.centos.noarch
openstack-nova-compute-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-gnocchi-carbonara-2.1.1-0.20160608165716.0bbf079.el7.centos.noarch
openstack-aodh-common-2.0.1-0.20160607164941.77238e3.el7.centos.noarch
openstack-manila-3.0.0-0.20160608190848.3dfc4dc.el7.centos.noarch
openstack-swift-proxy-2.7.1-0.20160608221859.11c5ef7.el7.centos.noarch
openstack-nova-api-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-swift-container-2.7.1-0.20160608221859.11c5ef7.el7.centos.noarch
python2-openstacksdk-0.8.7-0.20160609094719.a93ceb2.el7.centos.noarch
openstack-puppet-modules-8.0.0-0.20160520142355.6a3e8bf.el7.centos.noarch
openstack-gnocchi-api-2.1.1-0.20160608165716.0bbf079.el7.centos.noarch
openstack-neutron-metering-agent-9.0.0-0.20160609100034.4272cfc.el7.centos.noarch
openstack-aodh-evaluator-2.0.1-0.20160607164941.77238e3.el7.centos.noarch
openstack-heat-api-7.0.0-0.20160609065225.1e47fac.el7.centos.noarch
openstack-glance-13.0.0-0.20160609161924.551018a.el7.centos.noarch
openstack-swift-2.7.1-0.20160608221859.11c5ef7.el7.centos.noarch
openstack-tripleo-0.0.1-0.20160427104752.b076a5a.el7.centos.noarch
openstack-mistral-api-3.0.0-0.20160609153930.88b3994.el7.centos.noarch
openstack-nova-common-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-nova-conductor-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-nova-cells-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-sahara-api-5.0.0-0.20160609071340.165172f.el7.centos.noarch
openstack-neutron-9.0.0-0.20160609100034.4272cfc.el7.centos.noarch
openstack-gnocchi-common-2.1.1-0.20160608165716.0bbf079.el7.centos.noarch
openstack-tempest-12.0.1-0.20160608094412.ebe375c.el7.centos.noarch
python2-django-openstack-auth-2.3.1-0.20160609091907.d2b9c5a.el7.centos.noarch
openstack-neutron-openvswitch-9.0.0-0.20160609100034.4272cfc.el7.centos.noarch
openstack-aodh-expirer-2.0.1-0.20160607164941.77238e3.el7.centos.noarch
openstack-ironic-conductor-5.1.1-0.20160609093258.d58e2bc.el7.centos.noarch
openstack-heat-engine-7.0.0-0.20160609065225.1e47fac.el7.centos.noarch
openstack-ironic-inspector-3.3.1-0.20160607104948.457c127.el7.centos.noarch
openstack-ceilometer-api-6.0.1-0.20160608200830.726f243.el7.centos.noarch
openstack-ceilometer-common-6.0.1-0.20160608200830.726f243.el7.centos.noarch
openstack-swift-account-2.7.1-0.20160608221859.11c5ef7.el7.centos.noarch
openstack-keystone-10.0.0-0.20160609012819.eafa8b6.el7.centos.noarch
openstack-selinux-0.7.2-1.el7.noarch
openstack-mistral-executor-3.0.0-0.20160609153930.88b3994.el7.centos.noarch
openstack-nova-scheduler-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-swift-plugin-swift3-1.10-1.el7.noarch
openstack-nova-novncproxy-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-sahara-engine-5.0.0-0.20160609071340.165172f.el7.centos.noarch
openstack-heat-common-7.0.0-0.20160609065225.1e47fac.el7.centos.noarch
openstack-ironic-common-5.1.1-0.20160609093258.d58e2bc.el7.centos.noarch
openstack-cinder-9.0.0-0.20160608231321.0d2e9cc.el7.centos.noarch
puppet-openstack_extras-8.0-0.20160603174417.f9aebca.el7.centos.noarch
openstack-nova-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-tripleo-common-5.0.0-0.20160609081839.fbc0ee3.el7.centos.noarch
openstack-dashboard-10.0.0-0.20160609090113.ce68bf6.el7.centos.noarch
openstack-gnocchi-metricd-2.1.1-0.20160608165716.0bbf079.el7.centos.noarch
openstack-neutron-lbaas-9.0.0-0.20160609054819.22f5290.el7.centos.noarch
openstack-aodh-listener-2.0.1-0.20160607164941.77238e3.el7.centos.noarch
openstack-ironic-api-5.1.1-0.20160609093258.d58e2bc.el7.centos.noarch
openstack-heat-api-cloudwatch-7.0.0-0.20160609065225.1e47fac.el7.centos.noarch
openstack-ceilometer-central-6.0.1-0.20160608200830.726f243.el7.centos.noarch
openstack-ceilometer-compute-6.0.1-0.20160608200830.726f243.el7.centos.noarch
openstack-ceilometer-polling-6.0.1-0.20160608200830.726f243.el7.centos.noarch
openstack-zaqar-3.0.0-0.20160608232004.cb0e04a.el7.centos.noarch
openstack-nova-console-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-sahara-common-5.0.0-0.20160609071340.165172f.el7.centos.noarch
python-openstackclient-2.6.1-0.20160609034343.db0849e.el7.centos.noarch
openstack-gnocchi-statsd-2.1.1-0.20160608165716.0bbf079.el7.centos.noarch
openstack-neutron-ml2-9.0.0-0.20160609100034.4272cfc.el7.centos.noarch
openstack-aodh-notifier-2.0.1-0.20160607164941.77238e3.el7.centos.noarch
openstack-heat-api-cfn-7.0.0-0.20160609065225.1e47fac.el7.centos.noarch
openstack-ceilometer-collector-6.0.1-0.20160608200830.726f243.el7.centos.noarch
openstack-ceilometer-notification-6.0.1-0.20160608200830.726f243.el7.centos.noarch
openstack-tripleo-puppet-elements-5.0.0-0.20160607234408.3b2ddd9.el7.centos.noarch
openstack-mistral-engine-3.0.0-0.20160609153930.88b3994.el7.centos.noarch
openstack-nova-network-14.0.0-0.20160609152748.1249617.el7.centos.noarch
openstack-sahara-5.0.0-0.20160609071340.165172f.el7.centos.noarch
python-openstack-mistral-3.0.0-0.20160609153930.88b3994.el7.centos.noarch
openstack-tripleo-image-elements-5.0.0-0.20160607234521.3053ef4.el7.centos.noarch
openstack-tripleo-heat-templates-5.0.0-0.20160609160634.8027ae2.el7.centos.noarch
openstack-heat-templates-0.0.1-0.20160608123220.c9c5e15.el7.centos.noarch
openstack-neutron-common-9.0.0-0.20160609100034.4272cfc.el7.centos.noarch
openstack-gnocchi-indexer-sqlalchemy-2.1.1-0.20160608165716.0bbf079.el7.centos.noarch
openstack-aodh-api-2.0.1-0.20160607164941.77238e3.el7.centos.noarch
openstack-manila-share-3.0.0-0.20160608190848.3dfc4dc.el7.centos.noarch
openstack-utils-2015.2-1.el7.noarch
openstack-aodh-compat-2.0.1-0.20160607164941.77238e3.el7.centos.noarch
openstack-swift-object-2.7.1-0.20160608221859.11c5ef7.el7.centos.noarch
It is possible that this is configuration issue and that if the config values are more suited to the environment, we could complete deploy. If so, this would be a doc request.
Another trace:
[stack@undercloud ~]$ openstack baremetal introspection bulk start 42fc-4109- a6a5-de5bf80800 14
Setting nodes for introspection to manageable...
Starting introspection of node: 5e7af372-
Internal server error (ConnectionRefu sed): Error has occurred while handling request for http://<ip>:6385/ v1/nodes/ 5e7af372- 42fc-4109- a6a5-de5bf80800 14: HTTPConnectionP ool(host= '<ip>', port=6385): Max retries exceeded with url: /v1/nodes/ 5e7af372- 42fc-4109- a6a5-de5bf80800 14 (Caused by NewConnectionEr ror('<requests. packages. urllib3. connection. HTTPConnection object at 0x4456e10>: Failed to establish a new connection: [Errno 113] EHOSTUNREACH',))