[Dell PowerFlex] [scaleio]: password appears in plain text when creating a volume from an image

[Jean Pierre Roquesalane]

When creating a volume from an image, there is a step where the volume is attached.

During that operation, the password used by the PowerFlex driver to attach the volume from the backend appears in plain text in the logfile as shown here:

Jan 13 21:14:02 pflex153 cinder-volume[181019]: DEBUG oslo.privsep.daemon [-] privsep: reply[5c0c3eba-b6eb-493c-883a-a24282dcbc0e]: (4, '****ous') {{(pid=181842) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}

Note: the password has been masked in the example above.

[Sofia Enriquez (lsofia-enriquez)]

Hi Jean,
Is this happening also on generic-nfs or this is only happening with Dell PowerFlex?

Rajat suggested that this could also be a Nova bug, because also does some formatting while attaching the volume,  so I'm adding the project to the bug.

Cheers,

[Jean Pierre Roquesalane (jproque15130)]

This is powerflex issue as this the password used by our client driver to access the volume from the powerflex backend.

As this is PowerFlex, this is block-only.

[Jean Pierre Roquesalane (jproque15130)]

It seems that the password is displayed every time the driver tries to attach or detach a volume.

The following logs are display when executing an image to volume operation:
openstack volume create --size 32 --type powerflex --image cirros-0.5.2-x86_64-disk volpflex

When attaching!
Jan 25 10:32:26 zed-devstask cinder-volume[45668]: INFO cinder.volume.drivers.dell_emc.powerflex.driver [None req-bc7b1480-5cae-4fa3-b451-c2a36a79e285 admin None] Call os-brick to attach PowerFlex volume.
Jan 25 10:32:26 zed-devstask cinder-volume[45668]: DEBUG cinder.volume.drivers.dell_emc.powerflex.utils [None req-bc7b1480-5cae-4fa3-b451-c2a36a79e285 admin None] Converted id 2d59df0e-2ebc-4868-ab74-c9aa91a66fdf to PowerFlex OS name LVnfDi68SGirdMmqkaZv3w==. {{(pid=45668) id_to_base64 /opt/stack/cinder/cinder/volume/drivers/dell_emc/powerflex/utils.py:45}}
Jan 25 10:32:26 zed-devstask cinder-volume[45668]: DEBUG os_brick.initiator.connectors.scaleio [None req-bc7b1480-5cae-4fa3-b451-c2a36a79e285 admin None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.scaleio.ScaleIOConnector object at 0x7fc4e784b160>, 'connection_properties': {'scaleIO_volname': 'LVnfDi68SGirdMmqkaZv3w==', 'hostIP': None, 'serverIP': '10.225.108.171', 'serverPort': 443, 'serverUsername': 'admin', 'iopsLimit': None, 'bandwidthLimit': None, 'scaleIO_volume_id': '5aafd80d00000004', 'config_group': 'powerflex', 'failed_over': False, 'verify_certificate': False, 'certificate_path': None}}" {{(pid=45668) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:174}}
Jan 25 10:32:26 zed-devstask cinder-volume[45668]: DEBUG os_brick.initiator.connectors.base [None req-bc7b1480-5cae-4fa3-b451-c2a36a79e285 admin None] Acquiring lock "scaleio" by "os_brick.initiator.connectors.scaleio.ScaleIOConnector.connect_volume" {{(pid=45668) inner /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/base.py:68}}
Jan 25 10:32:26 zed-devstask cinder-volume[45668]: DEBUG os_brick.initiator.connectors.base [None req-bc7b1480-5cae-4fa3-b451-c2a36a79e285 admin None] Lock "scaleio" acquired by "os_brick.initiator.connectors.scaleio.ScaleIOConnector.connect_volume" :: waited 0.001s {{(pid=45668) inner /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/base.py:73}}
Jan 25 10:32:26 zed-devstask cinder-volume[45668]: INFO os_brick.initiator.connectors.scaleio [None req-bc7b1480-5cae-4fa3-b451-c2a36a79e285 admin None] Get ScaleIO connector password from configuration file
Jan 25 10:32:26 zed-devstask cinder-volume[45668]: DEBUG oslo.privsep.daemon [-] privsep: reply[7f6c8418-69aa-46d4-8595-4f9e02109658]: (4, 'Password123!') {{(pid=46568) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}

When detaching:

Jan 25 10:32:39 zed-devstask cinder-volume[45668]: INFO cinder.volume.drivers.dell_emc.powerflex.driver [None req-bc7b1480-5cae-4fa3-b451-c2a36a79e285 admin None] Call os-brick to detach PowerFlex volume.
Jan 25 10:32:39 zed-devstask cinder-volume[45668]: DEBUG cinder.volume.drivers.dell_emc.powerflex.utils [None req-bc7b1480-5cae-4fa3-b451-c2a36a79e285 admin None] Converted id 2d59df0e-2ebc-4868-ab74-c9aa91a66fdf to Power...

[Jean Pierre Roquesalane (jproque15130)]

This doesn't happen when attaching or detaching a volume with nova.

This has something to do between os_brick, privsep and cinder. I need to investigate on why this password appears in clear.

[Jean Pierre Roquesalane (jproque15130)]

The issue happens only when creating a volume from an image at the time it attaches or detaches the volume.

Seems there is something wrong between cinder and privsep called by os_brick.

Cinder calls os-brick:

https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/dell_emc/powerflex/driver.py#L1199

Then os-brick initiator makes a privilege call since it needs to access a restricted file

https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/scaleio.py#L90

It communicates with oslo_privsep to fetch the required information

https://github.com/openstack/os-brick/blob/master/os_brick/privileged/scaleio.py#L78

Then this is where the password is displayed in clear in the logs

https://github.com/openstack/oslo.privsep/blob/master/oslo_privsep/daemon.py#L501

[Jean Pierre Roquesalane (jproque15130)]

The issue happens only when creating a volume from an image at the time it attaches or detaches the volume.
Seems there is something wrong between cinder and privsep called by os_brick.

Cinder calls os-brick:
https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/dell_emc/powerflex/driver.py#L1199

Then os-brick initiator makes a privilege call since it needs to access a restricted file

https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/scaleio.py#L90

It communicates with oslo_privsep to fetch the required information

https://github.com/openstack/os-brick/blob/master/os_brick/privileged/scaleio.py#L78

Then this is where the password is displayed in clear in the logs

https://github.com/openstack/oslo.privsep/blob/master/oslo_privsep/daemon.py#L501

[Eric Harney (eharney)]

I think the fix for this is to introduce a new privsep entrypoint into os_brick/privileged/__init__.py and pass in a different logger_name which is configured to only log at ERROR level, which should avoid logging the output.

[Jean Pierre Roquesalane (jproque15130)]

Do you have any link to an example of such an implementation?

[Jeremy Stanley (fungi)]

Flagged this as a security hardening opportunity, class B3 in our taxonomy since it seems to only affect debug-level logging. https://security.openstack.org/vmt-process.html#report-taxonomy

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick 6.5.0 release.

[Bryan Neumann (bryanneumann)]

I propose adjusting the logging to remove the possibility of inadvertently logging sensitive data.

As this line, https://opendev.org/openstack/oslo.privsep/src/commit/f0c7eca61b9cbd3ae3dbcab7f29ca74dd0e2dc1d/oslo_privsep/daemon.py#L499, logs the reply from any command it was used to run, the issue could manifest elsewhere. To avoid this, the line performing the logging could be changed from

LOG.debug('privsep: reply[%(msgid)s]: %(reply)s',
   {'msgid': msgid, 'reply': reply})
to the following that will indicate True/False if there is a reply without printing it out.
 LOG.debug('privsep: reply[%(msgid)s]: %(reply)s',
                          {'msgid': msgid, 'reply?': reply != None})

This seems safer overall as the issue could show up for other drivers in the future.

[Bryan Neumann (bryanneumann)]

Changed project from Cinder to oslo.privsep