Tempest test test_list_get_baremetal_nodes_baremetal failed

[Sergii Turivnyi]

Detailed bug description:
ISO: http://paste.openstack.org/show/494203/
Configuration:
Compute - QEMU.
Network - Neutron with VLAN segmentation.
Storage Backends - Ceph RBD for volumes (Cinder), Ceph RBD for ephemeral volumes (Nova), Ceph RBD for images (Glance)
Additional services - Install Ironic

In tab Settings->Compute check Nova quotas

Nodes: controller, compute, ironic, ceph-osd

Steps to reproduce:
1. Deploy ISO in configuration see (Detailed bug description)
2. Navigate to controller node
3. Install git (use apt-get install git)
4. Clone script to deploy rally + tempest
   # git clone https://github.com/obutenko/mos-rally-verify.git
5. Navigate to the https://github.com/obutenko/mos-rally-verify
6. Execute necessary steps to deploy tempest
7. Tun test in debug mode
    #rally --debug verify start --regex tempest.api.compute.admin.test_baremetal_nodes

Expected results:
Test is passed

Actual result:
Test is Failed
(see comments)

Reproducibility:
See attachment

Workaround:
---

Impact:
---

Description of the environment:
See (Detailed bug description)

Additional information:
Error Message

test failed

Stacktrace

Traceback (most recent call last):
  File "tempest/api/compute/admin/test_baremetal_nodes.py", line 47, in test_list_get_baremetal_nodes
    self.client.list_baremetal_nodes()['nodes']])
  File "tempest/lib/services/compute/baremetal_nodes_client.py", line 31, in list_baremetal_nodes
    resp, body = self.get(url)
  File "tempest/lib/common/rest_client.py", line 274, in get
    return self.request('GET', url, extra_headers, headers)
  File "tempest/lib/common/rest_client.py", line 642, in request
    resp, resp_body)
  File "tempest/lib/common/rest_client.py", line 761, in _error_checker
    message=message)
tempest.lib.exceptions.ServerFault: Got server fault
Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'ironicclient.exc.AmbiguousAuthSystem'>

[Roman Podoliaka (rpodolyaka)]

Sergii, please provide logs and/or a diagnostic snapshot.

[Sergii Turivnyi (sturivnyi)]

Debug trace
http://paste.openstack.org/show/494979/

[Sergii Turivnyi (sturivnyi)]

Updated Bug Description
Added Debug trace

[Roman Podoliaka (rpodolyaka)]

Sergii, Tempest logs are not really useful either :( Please provide logs of openstack services. Why not simply use a diagnostic snapshot?

MOS Ironic team, could you please assist with this? (as you may have a ready environment). The error in Tempest logs indicates there was a problem with authentication:

https://github.com/openstack/python-ironicclient/blob/c6e9bfd6b1de8c7bd8103a0c58ca51bf8e56e85c/ironicclient/client.py#L124-L140

This looks like a configuration problem of either Ironic endpoints or ironicclient credentials in nova config.

[Sergii Turivnyi (sturivnyi)]

We (ironic team) have checked this case everything ok . Obviously this bug related to nova.

[Sergii Turivnyi (sturivnyi)]

All tests related to nodes is passed on this configuration
{0} tempest.api.baremetal.admin.test_nodes.TestNodes.test_create_node [0.865365s] ... ok
{0} tempest.api.baremetal.admin.test_nodes.TestNodes.test_delete_node [0.922796s] ... ok
{0} tempest.api.baremetal.admin.test_nodes.TestNodes.test_get_console [0.430309s] ... ok
{0} tempest.api.baremetal.admin.test_nodes.TestNodes.test_get_node_boot_device [0.366767s] ... ok
{0} tempest.api.baremetal.admin.test_nodes.TestNodes.test_get_node_by_instance_uuid [1.246326s] ... ok
{0} tempest.api.baremetal.admin.test_nodes.TestNodes.test_get_node_supported_boot_devices [0.257733s] ... ok
{0} tempest.api.baremetal.admin.test_nodes.TestNodes.test_list_nodes [0.488425s] ... ok
{0} tempest.api.baremetal.admin.test_nodes.TestNodes.test_list_nodes_association [1.655643s] ... ok
{0} tempest.api.baremetal.admin.test_nodes.TestNodes.test_node_port_list [0.474649s] ... ok
{0} tempest.api.baremetal.admin.test_nodes.TestNodes.test_node_port_list_no_ports [0.513818s] ... ok
{0} tempest.api.baremetal.admin.test_nodes.TestNodes.test_set_console_mode [0.529583s] ... ok
{0} tempest.api.baremetal.admin.test_nodes.TestNodes.test_set_node_boot_device [0.298091s] ... ok
{0} tempest.api.baremetal.admin.test_nodes.TestNodes.test_show_node [0.207494s] ... ok
{0} tempest.api.baremetal.admin.test_nodes.TestNodes.test_update_node [0.424459s] ... ok
{0} tempest.api.baremetal.admin.test_nodes.TestNodes.test_validate_driver_interface [0.211273s] ... ok

[Sergii Turivnyi (sturivnyi)]

Roman, diagnostic snapshot you can download here: http://www.ex.ua/437525489833

[Roman Podoliaka (rpodolyaka)]

Works for me on the latest ISO (#284):

 running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpqqpnRq
{0} tempest.api.compute.admin.test_baremetal_nodes.BaremetalNodesAdminTestJSON.test_list_get_baremetal_nodes [4.187784s] ... ok

Captured stderr:
~~~~~~~~~~~~~~~~
    /home/rally/.rally/tempest/for-deployment-84019a1e-62f3-43d5-b917-989520184f53/.venv/local/lib/python2.7/site-packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
      InsecurePlatformWarning
    /home/rally/.rally/tempest/for-deployment-84019a1e-62f3-43d5-b917-989520184f53/.venv/local/lib/python2.7/site-packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
      InsecurePlatformWarning
    /home/rally/.rally/tempest/for-deployment-84019a1e-62f3-43d5-b917-989520184f53/.venv/local/lib/python2.7/site-packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
      InsecurePlatformWarning
    /home/rally/.rally/tempest/for-deployment-84019a1e-62f3-43d5-b917-989520184f53/.venv/local/lib/python2.7/site-packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
      InsecurePlatformWarning
    /home/rally/.rally/tempest/for-deployment-84019a1e-62f3-43d5-b917-989520184f53/.venv/local/lib/python2.7/site-packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and ma...

[Oleksiy Butenko (obutenko)]

http://paste.openstack.org/show/496409/
test failed on ci
MOS ISO 289

[Oleksiy Butenko (obutenko)]

contact me for access to the environment where tests are failing.

[Sergii Turivnyi (sturivnyi)]

Can reproduce on ISO: http://paste.openstack.org/show/496878/

(.venv) root@node-3:~# rally verify start --regex tempest.api.compute.admin.test_baremetal_nodes
. . .
. . .
======
Totals
======
Ran: 1 tests in 19.0000 sec.
 - Passed: 0
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 1
Sum of execute time for each test: 1.8506 sec.

==============
Worker Balance
==============
 - Worker 0 (1 tests) => 0:00:01.850586

[Roman Podoliaka (rpodolyaka)]

The Tempest test fails and there is an appropriate error in nova-api.log:

2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/baremetal_nodes.py", line 105, in index
2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions icli = _get_ironic_client()
2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/baremetal_nodes.py", line 76, in _get_ironic_client
2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions icli = ironic_client.get_client(CONF.ironic.api_version, **kwargs)
2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/ironicclient/client.py", line 137, in get_client
2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions raise exc.AmbiguousAuthSystem(exception_msg)
2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions AmbiguousAuthSystem: Must provide Keystone credentials or user-defined endpoint and token
2016-05-12 12:10:20.328 31692 ERROR nova.api.openstack.extensions
2016-05-12 12:10:20.333 31692 INFO nova.api.openstack.wsgi [req-14504d54-3f40-41ac-9dbb-34e1dab3473f 2a91e632bd98451d92038cf000ba4a52 0fc3f4af3c9c4590ba06aa3467c80be2 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'ironicclient.exc.AmbiguousAuthSystem'>

which indicates that ironic credentials in nova.conf were not specified correctly.

At the same time nova.conf says:

[ironic]
admin_tenant_name=services
admin_username=ironic
api_endpoint=http://10.109.1.3:6385/v1
admin_password=wStYWdzAtHsXYZxmV4RJJiRt
admin_url=http://10.109.1.3:35357/v2.0

but in Guru Meditation Report we can see that these settings were ignored:

root@node-2:~# ps -ef | grep nova-api
root 18511 15637 0 12:15 pts/33 00:00:00 grep --color=auto nova-api
nova 30240 1 1 05:03 ? 00:04:19 /usr/bin/python /usr/bin/nova-api --log-file=/var/log/nova/nova-api.log --config-file=/etc/nova/nova.conf
nova 31692 30240 0 05:03 ? 00:02:12 /usr/bin/python /usr/bin/nova-api --log-file=/var/log/nova/nova-api.log --config-file=/etc/nova/nova.conf
nova 31693 30240 0 05:03 ? 00:02:12 /usr/bin/python /usr/bin/nova-api --log-file=/var/log/nova/nova-api.log --config-file=/etc/nova/nova.conf
nova 31720 30240 0 05:03 ? 00:00:19 /usr/bin/python /usr/bin/nova-api --log-file=/var/log/nova/nova-api.log --config-file=/etc/nova/nova.conf
nova 31721 30240 0 05:03 ? 00:00:19 /usr/bin/python /usr/bin/nova-api --log-file=/var/log/nova/nova-api.log --config-file=/etc/nova/nova.conf
root@node-2:~# kill -SIGUSR2 30240

root@node-2:~# grep -i...

[Roman Podoliaka (rpodolyaka)]

MOS Ironic, could you please take a look? ^

[Roman Podoliaka (rpodolyaka)]

Obviously, restart of nova-api on all controller helps, and that explains why I could not reproduce this on my environment (I did reconfigure nova-api for HTTPS and restarted it). Comments: https://bugs.launchpad.net/mos/+bug/1570819/comments/5 and https://bugs.launchpad.net/mos/+bug/1570819/comments/9

[Pavlo Shchelokovskyy (pshchelo)]

might that be another problem with puppets in case of SSL used? similar to https://bugs.launchpad.net/mos/+bug/1571610 but for nova-compute with Ironic driver

[Pavlo Shchelokovskyy (pshchelo)]

the current resolution is the following:

there is indeed an API extension in nova for baremetal nodes that currently proxies requests to Ironic (I was completely unaware of such thing until today...). There are even "nova baremetal-node-list" and such command in CLI.

This extension is using the [ironic] config section, but during deployment, this section is populated with correct Ironic service credentials when nova-api is already running. We need to modify our puppets that install nova-compute with ironic driver on controllers to trigger nova-api restart too.

[Vasyl Saienko (vsaienko)]

Pavlo: nova-api may not be configured on the node where nova-compute is started.
The proper way is to configure ironic section in nova:api puppet class https://github.com/openstack/puppet-nova/blob/master/manifests/api.pp

[Vasyl Saienko (vsaienko)]

I have opened a bug to puppet-ironic in upstream. https://bugs.launchpad.net/puppet-nova/+bug/1581040

[Timur Nurlygayanov (tnurlygayanov)]

Why it is in progress? Do we have a commits on review for MOS?

[Timur Nurlygayanov (tnurlygayanov)]

Commit on review in upstream https://review.openstack.org/#/c/317593/

[Serge Kovaleff (serge-kovaleff)]

The commit is merged in upstream.

[Andrey Shestakov (ashestakov)]

addressed in review https://review.openstack.org/#/c/315558/

[Andrey Shestakov (ashestakov)]

addressed in review for mitaka https://review.openstack.org/#/c/322636/

[Timur Nurlygayanov (tnurlygayanov)]

Tests passed on CI:
https://mirantis.testrail.com/index.php?/tests/view/6641334&group_by=cases:section_id&group_order=asc&group_id=23902

Status changed to Fix Released.

[Kyrylo Romanenko (kromanenko)]

Test passed on CI
https://mirantis.testrail.com/index.php?/tests/view/6641334&group_by=cases:section_id&group_order=asc&group_id=23902

[Oleksiy Butenko (obutenko)]

test pass on tempest ci
iso 420,424,427