"Secure live migration with QEMU-native TLS in nova"-guide misses essential config option

[Josephine Seifert]

- [x] This doc is inaccurate in this way: ______

I followed the guide to setup qemu native tls for live migration. After checking, that libvirt is able to use tls using tcpdump to listen on the port for tls, I also wanted to check that it works when I live migrate an instance. Apparently it didn't. But it used the port for unencrypted TCP [1].

After digging through documentation and code afterwards I found that in this code part: https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L1120

@staticmethod
    def _live_migration_uri(dest):
        uris = {
            'kvm': 'qemu+%(scheme)s://%(dest)s/system',
            'qemu': 'qemu+%(scheme)s://%(dest)s/system',
            'xen': 'xenmigr://%(dest)s/system',
            'parallels': 'parallels+tcp://%(dest)s/system',
        }
        dest = oslo_netutils.escape_ipv6(dest)

        virt_type = CONF.libvirt.virt_type
        # TODO(pkoniszewski): Remove fetching live_migration_uri in Pike
        uri = CONF.libvirt.live_migration_uri
        if uri:
            return uri % dest

        uri = uris.get(virt_type)
        if uri is None:
            raise exception.LiveMigrationURINotAvailable(virt_type=virt_type)

        str_format = {
            'dest': dest,
            'scheme': CONF.libvirt.live_migration_scheme or 'tcp',
        }
        return uri % str_format

the uri is calculated using the config parameter 'live_migration_scheme' or using the hard coded tcp parameter. Coming from the guide for qemu native tls, there was no hint that this config option needs to be set.

In fact without setting this 'live_migration_scheme' config option to tls, there is no way to see, that the live migration still uses the unencrypted tcp connection - one has to use tcpdump and listen for tcp or tls to recognize it. Neither in the logs nor in any debug output there is any hint that it is still unencrypted!

Thus I conclude there might be OpenStack deployments which are configured as the guide say but these config changes have no effect!

- [x] This is a doc addition request.

To fix this the config parameter 'live_migration_scheme' should be set to tls and maybe there should be a warning in the documentation, that without doing this, the traffic is still unencrypted.

- [ ] I have a fix to the document that I can paste below including example: input and output.

[1] without setting 'live_migration_scheme' in the nova.conf
$ tcpdump -i INTERFACE -n -X port 16509 and '(tcp[((tcp[12] & 0xf0) >> 2)] < 0x14 || tcp[((tcp[12] & 0xf0) >> 2)] > 0x17)'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on INTERFACE, link-type EN10MB (Ethernet), capture size 262144 bytes
17:10:56.387407 IP 192.168.70.101.50900 > 192.168.70.100.16509: Flags [P.], seq 304:6488, ack 285, win 502, options [nop,nop,TS val 424149655 ecr 1875309961], length 6184
 0x0000: 4500 185c ad05 4000 4006 677c c0a8 4665 E..\..@.@.g|..Fe
 0x0010: c0a8 4664 c6d4 407d a407 70a6 15ad 0a5a ..Fd..@}..p....Z
 0x0020: 8018 01f6 2669 0000 0101 080a 1948 0297 ....&i.......H..
 0x0030: 6fc6 f589 0000 1828 2000 8086 0000 0001 o......(........
 0x0040: 0000 012f 0000 0000 0000 0009 0000 0000 .../............
 0x0050: 0000 0001 0000 000f 6465 7374 696e 6174 ........destinat
 0x0060: 696f 6e5f 786d 6c00 0000 0007 0000 129b ion_xml.........
 0x0070: 3c64 6f6d 6169 6e20 7479 7065 3d27 6b76 <domain.type='kv
 0x0080: 6d27 3e0a 2020 3c6e 616d 653e 696e 7374 m'>...<name>inst
 0x0090: 616e 6365 2d30 3030 3032 6539 393c 2f6e ance-00002e99</n
 0x00a0: 616d 653e 0a20 203c 7575 6964 3e37 6532 ame>...<uuid>7e2
 0x00b0: 6364 3839 352d 6263 3765 2d34 6634 352d cd895-bc7e-4f45-
 0x00c0: 6166 6264 2d33 3732 3166 3735 6134 3064 afbd-3721f75a40d
 0x00d0: 383c 2f75 7569 643e 0a20 203c 6d65 7461 8</uuid>...<meta

[2] after setting 'live_migration_scheme = tls'
$ tcpdump -i INTERFACE -n -X port 16514 and '(tcp[((tcp[12] & 0xf0) >> 2)] > 0x13 && tcp[((tcp[12] & 0xf0) >> 2)] < 0x18)'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on INTERFACE, link-type EN10MB (Ethernet), capture size 262144 bytes
16:55:47.746851 IP 192.168.70.100.35620 > 192.168.70.101.16514: Flags [P.], seq 1849334708:1849334914, ack 3121294199, win 502, options [nop,nop,TS val 1874401351 ecr 423241020], length 206
 0x0000: 4500 0102 a605 4000 4006 85d6 c0a8 4664 E.....@.@.....Fd
 0x0010: c0a8 4665 8b24 4082 6e3a 9bb4 ba0b 2b77 ..Fe.$@.n:....+w
 0x0020: 8018 01f6 2f43 0000 0101 080a 6fb9 1847 ..../C......o..G
 0x0030: 193a 253c 1603 0100 c901 0000 c503 0360 .:%<...........`
 0x0040: 50e3 23f7 2ab9 57b2 47f5 e0e8 b550 014b P.#.*.W.G....P.K
 0x0050: 35c6 fe95 3475 cc66 9f81 fc0d 1646 ab00 5...4u.f.....F..
 0x0060: 0052 c02c c087 cca9 c0ad c00a c02b c086 .R.,.........+..
 0x0070: c0ac c009 c008 c030 c08b cca8 c014 c02f .......0......./

All in all: I can switch back and forth using TLS or unencrypted TCP via the different ports with just changing the config parameter 'live_migration_scheme'. All tests were done on an OpenStack Train deployment.

-----------------------------------
Release: 22.1.0.dev604 on 2019-03-11 17:38:39
SHA: 6e6d78d204dbf23d1ffaf232456fb2917b4f8372
Source: https://opendev.org/openstack/nova/src/doc/source/admin/secure-live-migration-with-qemu-native-tls.rst
URL: https://docs.openstack.org/nova/latest/admin/secure-live-migration-with-qemu-native-tls.html

[Jeremy Stanley (fungi)]

We discussed this at some length earlier today in the #openstack-nova IRC channel:

http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2021-03-16.log.html#t2021-03-16T14:31:57

Given that the information is already public, and the bug in this case is incomplete documentation (it fails to mention one of the two configuration options necessary to enable TLS), there's no need for us to keep an embargo on the report. Per the VMT's taxonomy, this is a class D report (misleading documentation), but may warrant publication of a security note:

https://security.openstack.org/vmt-process.html#incident-report-taxonomy

https://wiki.openstack.org/wiki/Security/Security_Note_Process

[Jeremy Stanley (fungi)]

Josephine: You should feel free to push your documentation patch in public to review.opendev.org as normal. Once the fix merges (and maybe gets backported to supported stable branches), we can entertain publication of a security note to help raise awareness for the benefit of anyone who may have followed the old instructions and not realized they were incomplete.

[Josephine Seifert (josei)]

I added a patch for the documentation:
https://review.opendev.org/c/openstack/nova/+/781030

[Jeremy Stanley (fungi)]

It looks like the last of the backports for https://review.opendev.org/q/Ia5130d411706bf7e1c983156158011a3bc6d5cd6 will merge soon, is there a security note draft somewhere? I would say given the pace at which we assign OSSN IDs, just go ahead and write it in https://wiki.openstack.org/w/index.php?title=OSSN/OSSN-0089&action=edit and we can announce it as OSSN-0089 once it's in good shape.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 23.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 21.2.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 22.2.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 20.6.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova stein-eol release.