get_hash_str() not working in FIPS mode

Bug #1801828 reported by Joshua Cornutt
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
In Progress
Medium
Joshua Cornutt

Bug Description

Description
===========
I run a Red Hat OpenStack 13 lab cloud in "forced" FIPS mode (fips=1 kernel flag) and this is one of the few issues I encountered along the way that required patching. Basically, MD5 is disabled in OpenSSL / Python's hashlib / NSS and anything else that honors FIPS mode, regardless of MD5's intended usage (for data security or not).

When creating an instance, MD5 is being used by privsep for get 7 random characters to append to the image cache filename.

Proposed patch - https://review.openstack.org/#/c/615704/

Steps to reproduce
==================
Launch an instance using CLI/API or Horizon.
(bootstrap-venv) [root@compute-1 bootstrap]# openstack server create \
    --image '54568928-d6bd-47b0-8985-faa34b8aefd2' \
    --flavor '63c23b65-6e9c-4910-887a-188a7c8b61da' \
    --key-name 'default' \
    --security-group '59c728b5-4856-4714-a4ae-21fcb239f8e1' \
    --network '8017a74e-de17-4abd-8261-55abad7fbf35' \
    'test-fips'

Expected result
===============
Instance gets created and enters the "running" state.

Actual result
=============
Instance sometimes gets stuck in "scheduling" state but other times fails with a Horizon error popup of "Error: Failed to perform requested operation on instance "test-fips", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 585f5f53-aa31-4e88-9a13-bc1d517e9d0f.]".

/var/log/nova/nova-compute.log shows a FIPS violation, stack trace -

2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [req-65312099-b42c-464e-a5d7-e46e0b07edc9 31a8c18286284697a9714cfe327f535f 3c3458e78f864f9385171fa621e5c757 - default default] [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] Instance failed to spawn: ValueError: error:060800A
3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] Traceback (most recent call last):
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2236, in _build_resources
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] yield resources
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2016, in _build_and_run_instance
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] block_device_info=block_device_info)
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3084, in spawn
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] block_device_info=block_device_info)
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3470, in _create_image
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] os_type_with_default)
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] File "/usr/lib/python2.7/site-packages/nova/virt/disk/api.py", line 97, in get_file_extension_for_os_type
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] return utils.get_hash_str(extension)[:7]
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] File "/usr/lib/python2.7/site-packages/nova/utils.py", line 1064, in get_hash_str
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] return hashlib.md5(base_str).hexdigest()
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]
2018-11-06 01:42:24.204 35447 INFO nova.compute.manager [req-65312099-b42c-464e-a5d7-e46e0b07edc9 31a8c18286284697a9714cfe327f535f 3c3458e78f864f9385171fa621e5c757 - default default] [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] Terminating instance

/var/log/nova/nova-conductor.log also shows a FIPS error -

2018-11-06 01:42:25.728 35493 ERROR nova.scheduler.utils [req-73ae3a5e-9936-4f1d-a45e-4dc9e145cd77 5e361d86f0fa4cfc82a1a0d481538d2c 0088c9e9ed484e41be132623e4a9e780 - a7052d336b4d45b196ab457dde9b34c6 a7052d336b4d45b196ab457dde9b34c6] [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] Error from last host: compute-1.us-east-1.nonplus.io (node compute-1.us-east-1.nonplus.io): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1825, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2105, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 585f5f53-aa31-4e88-9a13-bc1d517e9d0f was re-scheduled: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips\n']

Environment
===========

(bootstrap-venv) [root@compute-1 bootstrap]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.5 (Maipo)

(bootstrap-venv) [root@compute-1 bootstrap]# sysctl crypto.fips_enabled && cat /proc/sys/crypto/fips_enabled
crypto.fips_enabled = 1
1

(bootstrap-venv) [root@compute-1 bootstrap]# rpm -qa | grep nova
openstack-nova-api-17.0.5-3.d7864fbgit.el7ost.noarch
openstack-nova-compute-17.0.5-3.d7864fbgit.el7ost.noarch
openstack-nova-novncproxy-17.0.5-3.d7864fbgit.el7ost.noarch
openstack-nova-placement-api-17.0.5-3.d7864fbgit.el7ost.noarch
python2-novaclient-10.1.0-1.el7ost.noarch
openstack-nova-common-17.0.5-3.d7864fbgit.el7ost.noarch
openstack-nova-conductor-17.0.5-3.d7864fbgit.el7ost.noarch
python-nova-17.0.5-3.d7864fbgit.el7ost.noarch
openstack-nova-console-17.0.5-3.d7864fbgit.el7ost.noarch
openstack-nova-scheduler-17.0.5-3.d7864fbgit.el7ost.noarch

Tags: compute fips
Changed in nova:
assignee: nobody → Joshua Cornutt (oioooioi)
status: New → In Progress
melanie witt (melwitt)
Changed in nova:
importance: Undecided → Medium
tags: added: compute
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.