public rdo-cloud connection failures are causing bmc setup issues intermittently

[wes hayutin]

http://paste.openstack.org/show/727312/

Openstack Pastebin

posted on Aug 3, 2018, 9:47:22 PM

 130.141045] cloud-init[2067]: ++ neutron port-show bmc-43901_2 -c fixed_ips -f value
[ 130.310133] cloud-init[2067]: neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
[ 131.086258] cloud-init[2067]: parse error: Invalid numeric literal at line 1, column 15
[ 131.096157] cloud-init[2067]: + bmc_ip=
[ 131.097048] cloud-init[2067]: + '[' -z '' ']'
[ 131.109048] cloud-init[2067]: ++ neutron port-show bmc-43901_2 -c fixed_ips -f json
[ 131.111197] cloud-init[2067]: ++ jq -r '.fixed_ips[0].ip_address'
[ 131.285114] cloud-init[2067]: neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
[ 132.090700] cloud-init[2067]: + bmc_ip=10.0.1.15
[ 132.091789] cloud-init[2067]: + unit=openstack-bmc-baremetal-43901_2.service
[ 132.093090] cloud-init[2067]: + cat
[ 132.094942] cloud-init[2067]: + echo ' - ip_netmask: 10.0.1.15/'
[ 132.096154] cloud-init[2067]: + for i in '$(seq 1 4)'
[ 132.097139] cloud-init[2067]: + bm_port=baremetal-43901_3
[ 132.109056] cloud-init[2067]: ++ neutron port-show baremetal-43901_3 -c device_id -f value
[ 132.280144] cloud-init[2067]: neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
[ 133.139439] cloud-init[2067]: + bm_instance=cde5b0bc-6920-4a06-9bc3-1951d89ff18d
[ 133.140972] cloud-init[2067]: + bmc_port=bmc-43901_3
[ 133.153080] cloud-init[2067]: ++ neutron port-show bmc-43901_3 -c fixed_ips -f value
[ 133.155115] cloud-init[2067]: ++ jq -r .ip_address
[ 133.313101] cloud-init[2067]: neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
[ 134.243231] cloud-init[2067]: parse error: Invalid numeric literal at line 1, column 15
[ 134.253075] cloud-init[2067]: + bmc_ip=
[ 134.253848] cloud-init[2067]: + '[' -z '' ']'
[ 134.266071] cloud-init[2067]: ++ neutron port-show bmc-43901_3 -c fixed_ips -f json
[ 134.268336] cloud-init[2067]: ++ jq -r '.fixed_ips[0].ip_address'
[ 134.431102] cloud-init[2067]: neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
[ 135.450729] cloud-init[2067]: + bmc_ip=10.0.1.13
[ 135.452112] cloud-init[2067]: + unit=openstack-bmc-baremetal-43901_3.service
[ 135.453672] cloud-init[2067]: + cat
[ 135.455616] cloud-init[2067]: + echo ' - ip_netmask: 10.0.1.13/'
[ 135.456795] cloud-init[2067]: + systemctl enable config-bmc-ips
[ 135.461642] cloud-init[2067]: Created symlink from /etc/systemd/system/multi-user.target.wants/config-bmc-ips.service to /usr/lib/systemd/system/config-bmc-ips.service.
[ 135.508054] cloud-init[2067]: ++ seq 1 4
[ 135.509015] cloud-init[2067]: + for i in '$(seq 1 4)'
[ 135.509974] cloud-init[2067]: + bm_port=baremetal-43901_0
[ 135.510974] cloud-init[2067]: + unit=openstack-bmc-baremetal-43901_0.service
[ 135.512238] cloud-init[2067]: + systemctl enable openstack-bmc-baremetal-43901_0.service
[ 135.516128] cloud-init[2067]: Created symlink from /etc/systemd/system/multi-user.target.wants/openstack-bmc-baremetal-43901_0.service to /usr/lib/systemd/system/openstack-bmc-baremetal-43901_0.service.
[ 135.548686] cloud-init[2067]: + systemctl start openstack-bmc-baremetal-43901_0.service
         Starting config-bmc-ips Service...
[ 135.741066] os-net-config[2274]: [2018/08/03 07:59:52 PM] [INFO] Using config file at: /etc/os-net-config/config.yaml
[ 135.744155] os-net-config[2274]: [2018/08/03 07:59:52 PM] [INFO] Using mapping file at: /etc/os-net-config/mapping.yaml
[ 135.745952] os-net-config[2274]: [2018/08/03 07:59:52 PM] [INFO] Ifcfg net config provider created.
[ 135.747492] os-net-config[2274]: Traceback (most recent call last):
[ 135.748622] os-net-config[2274]: File "/bin/os-net-config", line 10, in <module>
[ 135.749907] os-net-config[2274]: sys.exit(main())
[ 135.750802] os-net-config[2274]: File "/usr/lib/python2.7/site-packages/os_net_config/cli.py", line 184, in main
[ 135.752965] os-net-config[2274]: obj = objects.object_from_json(iface_json)
[ 135.754201] os-net-config[2274]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 38, in object_from_json
[ 135.756376] os-net-config[2274]: return Interface.from_json(json)
[ 135.757527] os-net-config[2274]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 305, in from_json
[ 135.759359] os-net-config[2274]: opts = _BaseOpts.base_opts_from_json(json)
[ 135.760588] os-net-config[2274]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 256, in base_opts_from_json
[ 135.762520] os-net-config[2274]: addresses.append(Address.from_json(address))
[ 135.763768] os-net-config[2274]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 175, in from_json
[ 135.765565] os-net-config[2274]: return Address(ip_netmask)
[ 135.766585] os-net-config[2274]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 166, in __init__
[ 135.768742] os-net-config[2274]: ip_nw = netaddr.IPNetwork(self.ip_netmask)
[ 135.769987] os-net-config[2274]: File "/usr/lib/python2.7/site-packages/netaddr/ip/__init__.py", line 933, in __init__
[[1;31mFAILED[0m] Failed to start config-bmc-ips Service.
See 'systemctl status config-bmc-ips.service' for details.
[[1;33mDEPEND[0m] Dependency failed for openstack-bmc baremetal-43901_0 Service.
[ 135.786376] cloud-init[2067]: A dependency job for openstack-bmc-baremetal-43901_0.service failed. See 'journalctl -xe' for details.
[ 135.788373] cloud-init[2067]: + systemctl status openstack-bmc-baremetal-43901_0.service
[ 135.789780] os-net-config[2274]: raise AddrFormatError('invalid IPNetwork %s' % addr)
[ 135.791082] os-net-config[2274]: netaddr.core.AddrFormatError: invalid IPNetwork 10.0.1.11/
[ 135.799260] cloud-init[2067]: ��� openstack-bmc-baremetal-43901_0.service - openstack-bmc baremetal-43901_0 Service
[ 135.801063] cloud-init[2067]: Loaded: loaded (/usr/lib/systemd/system/openstack-bmc-baremetal-43901_0.service; enabled; vendor preset: disabled)
[ 135.803161] cloud-init[2067]: Active: inactive (dead)
[ 135.804126] cloud-init[2067]: Aug 03 19:59:52 bmc-43901 systemd[1]: Dependency failed for openstack-bmc baremetal-43901_0 Service.
[ 135.806060] cloud-init[2067]: Aug 03 19:59:52 bmc-43901 systemd[1]: Job openstack-bmc-baremetal-43901_0.service/start failed with result 'dependency'.
[ 135.808196] cloud-init[2067]: + for i in '$(seq 1 4)'
[ 135.809159] cloud-init[2067]: + bm_port=baremetal-43901_1
[ 135.811769] cloud-init[2067]: + unit=openstack-bmc-baremetal-43901_1.service
[ 135.813040] cloud-init[2067]: + systemctl enable openstack-bmc-baremetal-43901_1.service
[ 135.822580] cloud-init[2067]: Created symlink from /etc/systemd/system/multi-user.target.wants/openstack-bmc-baremetal-43901_1.service to /usr/lib/systemd/system/openstack-bmc-baremetal-43901_1.service.
[ 135.850137] cloud-init[2067]: + systemctl start openstack-bmc-baremetal-43901_1.service
         Starting config-bmc-ips Service...
[ 136.020065] os-net-config[2293]: [2018/08/03 07:59:52 PM] [INFO] Using config file at: /etc/os-net-config/config.yaml
[ 136.021990] os-net-config[2293]: [2018/08/03 07:59:52 PM] [INFO] Using mapping file at: /etc/os-net-config/mapping.yaml
[ 136.023791] os-net-config[2293]: [2018/08/03 07:59:52 PM] [INFO] Ifcfg net config provider created.
[ 136.025346] os-net-config[2293]: Traceback (most recent call last):
[ 136.026493] os-net-config[2293]: File "/bin/os-net-config", line 10, in <module>
[ 136.027787] os-net-config[2293]: sys.exit(main())
[ 136.028692] os-net-config[2293]: File "/usr/lib/python2.7/site-packages/os_net_config/cli.py", line 184, in main
[ 136.030790] os-net-config[2293]: obj = objects.object_from_json(iface_json)
[ 136.032031] os-net-config[2293]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 38, in object_from_json
[ 136.033873] os-net-config[2293]: return Interface.from_json(json)
[ 136.034960] os-net-config[2293]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 305, in from_json
[ 136.036727] os-net-config[2293]: opts = _BaseOpts.base_opts_from_json(json)
[ 136.037942] os-net-config[2293]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 256, in base_opts_from_json
[ 136.039883] os-net-config[2293]: addresses.append(Address.from_json(address))
[[1;31mFAILED[0m] Failed to start config-bmc-ips Service.
See 'systemctl status config-bmc-ips.service' for details.
[[1;33mDEPEND[0m] Dependency failed for openstack-bmc baremetal-43901_1 Service.
[ 136.053473] os-net-config[2293]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 175, in from_json
[ 136.055249] os-net-config[2293]: return Address(ip_netmask)
[ 136.056216] os-net-config[2293]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 166, in __init__
[ 136.057908] os-net-config[2293]: ip_nw = netaddr.IPNetwork(self.ip_netmask)
[ 136.059062] os-net-config[2293]: File "/usr/lib/python2.7/site-packages/netaddr/ip/__init__.py", line 933, in __init__
[ 136.060737] os-net-config[2293]: raise AddrFormatError('invalid IPNetwork %s' % addr)
[ 136.062024] os-net-config[2293]: netaddr.core.AddrFormatError: invalid IPNetwork 10.0.1.11/
[ 136.063501] cloud-init[2067]: A dependency job for openstack-bmc-baremetal-43901_1.service failed. See 'journalctl -xe' for details.
[ 136.069463] cloud-init[2067]: + systemctl status openstack-bmc-baremetal-43901_1.service
[ 136.070939] cloud-init[2067]: ��� openstack-bmc-baremetal-43901_1.service - openstack-bmc baremetal-43901_1 Service
[ 136.072642] cloud-init[2067]: Loaded: loaded (/usr/lib/systemd/system/openstack-bmc-baremetal-43901_1.service; enabled; vendor preset: disabled)
[ 136.074694] cloud-init[2067]: Active: inactive (dead)
[ 136.075627] cloud-init[2067]: + for i in '$(seq 1 4)'
[ 136.076559] cloud-init[2067]: + bm_port=baremetal-43901_2
[ 136.077520] cloud-init[2067]: + unit=openstack-bmc-baremetal-43901_2.service
[ 136.088223] cloud-init[2067]: + systemctl enable openstack-bmc-baremetal-43901_2.service
[ 136.089709] cloud-init[2067]: Created symlink from /etc/systemd/system/multi-user.target.wants/openstack-bmc-baremetal-43901_2.service to /usr/lib/systemd/system/openstack-bmc-baremetal-43901_2.service.
[ 136.115691] cloud-init[2067]: + systemctl start openstack-bmc-baremetal-43901_2.service
         Starting config-bmc-ips Service...
[ 136.283057] os-net-config[2312]: [2018/08/03 07:59:52 PM] [INFO] Using config file at: /etc/os-net-config/config.yaml
[ 136.284962] os-net-config[2312]: [2018/08/03 07:59:52 PM] [INFO] Using mapping file at: /etc/os-net-config/mapping.yaml
[ 136.286774] os-net-config[2312]: [2018/08/03 07:59:52 PM] [INFO] Ifcfg net config provider created.
[ 136.288311] os-net-config[2312]: Traceback (most recent call last):
[ 136.289434] os-net-config[2312]: File "/bin/os-net-config", line 10, in <module>
[ 136.290706] os-net-config[2312]: sys.exit(main())
[ 136.291584] os-net-config[2312]: File "/usr/lib/python2.7/site-packages/os_net_config/cli.py", line 184, in main
[ 136.293500] os-net-config[2312]: obj = objects.object_from_json(iface_json)
[ 136.294746] os-net-config[2312]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 38, in object_from_json
[ 136.296595] os-net-config[2312]: return Interface.from_json(json)
[ 136.297671] os-net-config[2312]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 305, in from_json
[ 136.299460] os-net-config[2312]: opts = _BaseOpts.base_opts_from_json(json)
[ 136.300665] os-net-config[2312]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 256, in base_opts_from_json
[ 136.302561] os-net-config[2312]: addresses.append(Address.from_json(address))
[ 136.303800] [[1;31mFAILED[0m] Failed to start config-bmc-ips Service.
See 'systemctl status config-bmc-ips.service' for details.
[[1;33mDEPEND[0m] Dependency failed for openstack-bmc baremetal-43901_2 Service.
os-net-config[2312]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 175, in from_json
[ 136.315989] os-net-config[2312]: return Address(ip_netmask)
[ 136.316930] os-net-config[2312]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 166, in __init__
[ 136.318656] os-net-config[2312]: ip_nw = netaddr.IPNetwork(self.ip_netmask)
[ 136.319740] os-net-config[2312]: File "/usr/lib/python2.7/site-packages/netaddr/ip/__init__.py", line 933, in __init__
[ 136.321416] os-net-config[2312]: raise AddrFormatError('invalid IPNetwork %s' % addr)
[ 136.322685] os-net-config[2312]: netaddr.core.AddrFormatError: invalid IPNetwork 10.0.1.11/
[ 136.324184] cloud-init[2067]: A dependency job for openstack-bmc-baremetal-43901_2.service failed. See 'journalctl -xe' for details.
[ 136.329961] cloud-init[2067]: + systemctl status openstack-bmc-baremetal-43901_2.service
[ 136.331448] cloud-init[2067]: ��� openstack-bmc-baremetal-43901_2.service - openstack-bmc baremetal-43901_2 Service
[ 136.333157] cloud-init[2067]: Loaded: loaded (/usr/lib/systemd/system/openstack-bmc-baremetal-43901_2.service; enabled; vendor preset: disabled)
[ 136.335208] cloud-init[2067]: Active: inactive (dead)
[ 136.336153] cloud-init[2067]: + for i in '$(seq 1 4)'
[ 136.337094] cloud-init[2067]: + bm_port=baremetal-43901_3
[ 136.338086] cloud-init[2067]: + unit=openstack-bmc-baremetal-43901_3.service
[ 136.339297] cloud-init[2067]: + systemctl enable openstack-bmc-baremetal-43901_3.service
[ 136.350291] cloud-init[2067]: Created symlink from /etc/systemd/system/multi-user.target.wants/openstack-bmc-baremetal-43901_3.service to /usr/lib/systemd/system/openstack-bmc-baremetal-43901_3.service.
[ 136.375817] cloud-init[2067]: + systemctl start openstack-bmc-baremetal-43901_3.service
         Starting config-bmc-ips Service...
[ 136.547061] os-net-config[2331]: [2018/08/03 07:59:53 PM] [INFO] Using config file at: /etc/os-net-config/config.yaml
[ 136.548982] os-net-config[2331]: [2018/08/03 07:59:53 PM] [INFO] Using mapping file at: /etc/os-net-config/mapping.yaml
[ 136.550760] os-net-config[2331]: [2018/08/03 07:59:53 PM] [INFO] Ifcfg net config provider created.
[ 136.552285] os-net-config[2331]: Traceback (most recent call last):
[ 136.553412] os-net-config[2331]: File "/bin/os-net-config", line 10, in <module>
[ 136.554694] os-net-config[2331]: sys.exit(main())
[ 136.555586] os-net-config[2331]: File "/usr/lib/python2.7/site-packages/os_net_config/cli.py", line 184, in main
[ 136.557660] os-net-config[2331]: obj = objects.object_from_json(iface_json)
[ 136.558912] os-net-config[2331]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 38, in object_from_json
[ 136.560799] os-net-config[2331]: return Interface.from_json(json)
[ 136.561891] os-net-config[2331]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 305, in from_json
[ 136.563696] os-net-config[2331]: opts = _BaseOpts.base_opts_from_json(json)
[ 136.564934] os-net-config[2331]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 256, in base_opts_from_json
[ 136.566849] os-net-config[2331]: addresses.append(Address.from_json(address))
[[1;31mFAILED[0m] Failed to start config-bmc-ips Service.
See 'systemctl status config-bmc-ips.service' for details.
[[1;33mDEPEND[0m] Dependency failed for openstack-bmc baremetal-43901_3 Service.
[ 136.578346] os-net-config[2331]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 175, in from_json
[ 136.580149] os-net-config[2331]: return Address(ip_netmask)
[ 136.581103] os-net-config[2331]: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 166, in __init__
[ 136.582891] os-net-config[2331]: ip_nw = netaddr.IPNetwork(self.ip_netmask)
[ 136.584058] os-net-config[2331]: File "/usr/lib/python2.7/site-packages/netaddr/ip/__init__.py", line 933, in __init__
[ 136.585718] os-net-config[2331]: raise AddrFormatError('invalid IPNetwork %s' % addr)
[ 136.586936] os-net-config[2331]: netaddr.core.AddrFormatError: invalid IPNetwork 10.0.1.11/
[ 136.588423] cloud-init[2067]: A dependency job for openstack-bmc-baremetal-43901_3.service failed. See 'journalctl -xe' for details.
[ 136.595252] cloud-init[2067]: + systemctl status openstack-bmc-baremetal-43901_3.service
[ 136.596864] cloud-init[2067]: ��� openstack-bmc-baremetal-43901_3.service - openstack-bmc baremetal-43901_3 Service
[ 136.598615] cloud-init[2067]: Loaded: loaded (/usr/lib/systemd/system/openstack-bmc-baremetal-43901_3.service; enabled; vendor preset: disabled)
[ 136.600732] cloud-init[2067]: Active: inactive (dead)
[ 136.601703] cloud-init[2067]: 2018-08-03 19:59:53,207 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [3]
[ 136.605788] cloud-init[2067]: 2018-08-03 19:59:53,218 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
[ 136.608150] cloud-init[2067]: 2018-08-03 19:59:53,218 - util.py[WARNING]: Running scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
ci-info: ++++++++++++++++Authorized keys from /home/centos/.ssh/authorized_keys for user centos++++++++++++++++

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/588678

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/588678
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=baa78023d076156d8214997eb88d28b9a2d8c1ab
Submitter: Zuul
Branch: master

commit baa78023d076156d8214997eb88d28b9a2d8c1ab
Author: Wes Hayutin <email address hidden>
Date: Fri Aug 3 18:59:24 2018 -0400

    update rdocloud dns settings to outside provider

    Closes-Bug: #1785342
    Change-Id: Icc238297e9af33adf74720700b20e797749882eb

[wes hayutin (weshayutin)]

This patch did not improve the situation
https://logs.rdoproject.org/45/560445/106/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/846c368/logs/undercloud/etc/resolv.conf.txt.gz

The above is a no-op change while using the new dns settings

[wes hayutin (weshayutin)]

We need to get this bug resolved https://bugs.launchpad.net/tripleo/+bug/1785055
to properly debug the issue at this point.

[wes hayutin (weshayutin)]

DNS changes did not resolve this issue...

From time to time we still see the following on the bmc console.

4] os-net-config[3803]: [2018/08/06 02:21:10 PM] NFO] No changes required for interface: eth0
[ 3997.497814] openstackbmc[3814]: No handlers could be found for logger "keystoneauth.identity.generic.base"
[ 3997.502214] openstackbmc[3814]: Traceback (most recent call last):
[ 3997.503420] openstackbmc[3814]: File "/usr/local/bin/openstackbmc", line 310, in <module>
[ 3997.504883] openstackbmc[3814]: main()
[ 3997.505679] openstackbmc[3814]: File "/usr/local/bin/openstackbmc", line 305, in main
[ 3997.507110] openstackbmc[3814]: os_cloud=args.os_cloud)
[ 3997.508474] openstackbmc[3814]: File "/usr/local/bin/openstackbmc", line 65, in __init__
[ 3997.509976] openstackbmc[3814]: cloud=os_cloud)
[ 3997.510895] openstackbmc[3814]: File "/usr/lib/python2.7/site-packages/os_client_config/__init__.py", line 86, in make_client
[ 3997.512878] openstackbmc[3814]: return cloud.get_legacy_client(service_key, constructor)
[ 3997.514367] openstackbmc[3814]: File "/usr/lib/python2.7/site-packages/os_client_config/cloud_config.py", line 397, in get_legacy_client
[ 3997.516460] openstackbmc[3814]: service_key, min_version=min_version, max_version=max_version)
[ 3997.517982] openstackbmc[3814]: File "/usr/lib/python2.7/site-packages/os_client_config/cloud_config.py", line 336, in get_session_endpoint
[ 3997.520299] openstackbmc[3814]: endpoint = self._get_highest_endpoint(service_types, kwargs)
[ 3997.521816] openstackbmc[3814]: File "/usr/lib/python2.7/site-packages/os_client_config/cloud_config.py", line 294, in _get_highest_endpoint
[ 3997.523954] openstackbmc[3814]: return session.get_endpoint(**kwargs)
[ 3997.525149] openstackbmc[3814]: File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 942, in get_endpoint
[ 3997.527089] openstackbmc[3814]: return auth.get_endpoint(self, **kwargs)
[ 3997.528407] openstackbmc[3814]: File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 379, in get_endpoint
[ 3997.530742] openstackbmc[3814]: allow_version_hack=allow_version_hack, **kwargs)
[ 3997.532114] openstackbmc[3814]: File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 270, in get_endpoint_data
[ 3997.534191] openstackbmc[3814]: service_catalog = self.get_access(session).service_catalog
[ 3997.535684] openstackbmc[3814]: File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
[ 3997.537650] openstackbmc[3814]: self.auth_ref = self.get_auth_ref(session)
[ 3997.538914] openstackbmc[3814]: File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 201, in get_auth_ref
[ 3997.541321] openstackbmc[3814]: return self._plugin.get_auth_ref(session, **kwargs)
[ 3997.542745] openstackbmc[3814]: File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py", line 63, in get_auth_ref
[ 3997.544685] openstackbmc[3814]: authenticated=False, log=False)
[ 3997.545811] openstackbmc[3814]: File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 848, in post
[ 3997.547633] openstackbmc[3814]: return self.request(url, 'POST', **...

[wes hayutin (weshayutin)]

last paste cut off the most important part

[ 3997.553918] openstackbmc[3814]: raise exceptions.ConnectFailure(msg)
[ 3997.555119] openstackbmc[3814]: keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to https://phx2.cloud.rdoproject.org:13000/v2.0/tokens: HTTPSConnectionPool(host='phx2.cloud.rdoproject.org', port=13000): Max retries exceeded with url: /v2.0/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x39b2a50>: Failed to establish a new connection: rrno -2] Name or service not known',))

[Alex Schultz (alex-schultz)]

This was previously diagnosed as issues with stuck stacks causing bmc not to properly bring up all the assigned network interfaces. Dropping to High as we should solve for that and monitor but it is an infrastructure related issue and not code related.

[Juan Antonio Osorio Robles (juan-osorio-robles)]

Is this still an issue?

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-quickstart-extras 2.1.1 release.