swap volume operation may leak credentials into debug logs

[Matt Riedemann]

The swap volume code in the compute service logs old and new volume connection_info dicts to debug here:

https://github.com/openstack/nova/blob/1ad41c0b0c844b65424251dbc3399039789b064f/nova/compute/manager.py#L4930

The new connection_info comes from Cinder:

https://github.com/openstack/nova/blob/1ad41c0b0c844b65424251dbc3399039789b064f/nova/compute/manager.py#L4901

That's a plain dict from the response which may contain credentials.

The old connection_info comes from the nova.objects.block_device.BlockDeviceMapping object, which uses SensitiveStringField to sanitize the field's value when logged:

https://github.com/openstack/nova/blob/1ad41c0b0c844b65424251dbc3399039789b064f/nova/compute/manager.py#L4904

https://github.com/openstack/oslo.versionedobjects/blob/1.23.0/oslo_versionedobjects/fields.py#L280

The new connection_info could contain credentials though, so we should mask those when logging it, even at debug level.

[Tristan Cacqueray (tristan-cacqueray)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Tristan Cacqueray (tristan-cacqueray)]

If this is only happening at DEBUG level, then it's probably a class B3 according to VMT's taxonomy: https://security.openstack.org/vmt-process.html#incident-report-taxonomy

[Matt Riedemann (mriedem)]

Can I post a patch for this in Gerrit or do I need to hold off for now? I can attach a .patch file in here if needed.

[Jeremy Stanley (fungi)]

I agree with Tristan's assessment that the VMT won't pursue an advisory in this case, per our long-standing policy about information leaks in DEBUG-level logging, thus there's no need to maintain the embargo and fixes can definitely be reviewed in public. I'm lifting the embargo now and switching this to a potential hardening opportunity.

[Matt Riedemann (mriedem)]

Here is an example from the logs in a CI run:

http://logs.openstack.org/94/465794/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/688500a/logs/screen-n-cpu.txt.gz#_May_19_22_46_34_664553

{
   u'driver_volume_type':u'iscsi',
   'connector':{
      'platform':'x86_64',
      'host':'ubuntu-xenial-internap-mtl01-8904270',
      'do_local_attach':False,
      'ip':'198.72.124.18',
      'os_type':'linux2',
      'multipath':False,
      'initiator': u'iqn.1993-08.org.debian:01:f5ecd8cdb33f'
   },
   'serial':u'8ab09912-5e7b-4ca4-988b-e5993bfe24f6',
   u'data':{
      u'access_mode':u'rw',
      u'target_discovered':False,
      u'encrypted':False,
      u'qos_specs':None,
      u'target_iqn': u'iqn.2010-10.org.openstack:volume-8ab09912-5e7b-4ca4-988b-e5993bfe24f6',
      u'target_portal': u'198.72.124.18:3260 ', u' volume_id':u'8ab09912-5e7b-4ca4-988b-e5993bfe24f6',
      u'target_lun':1,
      u'auth_password':u'y2UWZPbTLf27A3xV',
      u'auth_username':u'qAAHKzWja6X2yXRBwCym',
      u'auth_method':u'CHAP'
   }
}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/466533

[Matt Riedemann (mriedem)]

This is also a problem in python-cinderclient which logs the response body:

https://github.com/openstack/python-cinderclient/blob/84346b5dba784bfeb3a53ae83d400ba264263cf6/cinderclient/apiclient/client.py#L141

[Matt Riedemann (mriedem)]

Actually cinderclient should be fixed:

https://review.openstack.org/#/c/395119/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/466533
Reason: Looks like it was fixed under a different bug https://review.openstack.org/#/c/558694/