poor random seed generation resulting in duplicate random MAC generation for virbr0

[Trent Lloyd]

It was encountered in production that two systems bootstrapped at the same time with identical configurations generated the same random MAC address for virbr0-nic/virbr0

Looking at libvirt, the MAC address is created in src/util/virmacaddr.c:virMacAddrGenerate() using virRandomBits()

src/util/virrandom.c:virRandomOnceInit seeds the random number generator using this formula:
  unsigned int seed = time(NULL) ^ getpid();

This seems to be a popular method after a quick google but it's easy to see how this can be problematic. The time is only in seconds, and during boot of a relatively identical system these numbers are both likely to be relatively similar across multiple systems which is quite likely in cloud-like environments. Secondly, by using bitwise OR only a small difference is created and if the 1st or 2nd MSB of the pid or time are 0 then it would be easy to have colliding values.

Though problematic from basic logic, I also tested this with a small test program trying 67,921 unique combinations of time() and pid() which produced only 5,693 random seeds using PID range 6799-6810 and time() range 1502484340 to 1502489999.

During the actual incident in question, the 4 systems were all booted within 1-2 seconds of each other. We can see from dmesg that the two systems that generated the same MAC did in fact boot during the same second and the other two did not

[kvm01] Aug 11 16:45:47 unassigned kernel: [ 3.643820] rtc_cmos 00:00: setting system clock to 2017-08-11 20:45:42 UTC (1502484342)
[kvm02] Aug 11 16:45:45 unassigned kernel: [ 3.629268] rtc_cmos 00:00: setting system clock to 2017-08-11 20:45:40 UTC (1502484340)
[kvm03] Aug 11 16:45:45 unassigned kernel: [ 3.623670] rtc_cmos 00:00: setting system clock to 2017-08-11 20:45:40 UTC (1502484340)
[kvm04] Aug 11 16:45:48 unassigned kernel: [ 3.631698] rtc_cmos 00:00: setting system clock to 2017-08-11 20:45:43 UTC (1502484343)

We can also see the libvirt pid happened to be identical on the 2 systems
Aug 11 18:11:15 kvm01 libvirtd[13209]: libvirt version: 1.3.1
Aug 11 18:11:15 kvm02 libvirtd[6409]: libvirt version: 1.3.1
Aug 11 18:11:15 kvm03 libvirtd[6409]: libvirt version: 1.3.1
Aug 11 18:11:17 kvm04 libvirtd[6472]: libvirt version: 1.3.1

Leading to the same MAC being generated. The practical fall out of this was that the virbr0-nic MAC was used as a unique identifier by MAAS which caused problems with it thinking the two hosts were the same.

Given deployments of libvirt-bin in bulk at a similar time are not uncommon, and that having similar PIDs at bootup is also not all that unlikely - I think the randomness of the libvirt seed needs to be improved to prevent this happening on another occasion.

I think there are also potentially wider affects. The same random number generator is used to generate among other things MACs for virtual machines, machine UUIDs, qemuDomainSecretAESSetup (if GNUTLS is not in use, which it maybe is not), etc. Brute forcing the initial random seed in an attempt to generate duplicate or predict UUIDs or MACs may be difficult to target in practice but probably not impossible - perhaps if you could force reboot two systems at the same time, etc.

It seems this has been observed at least once before, where many VMs got the same MAC. But the current upstream source is the same as that of xenial:libvirt where the issue was observed.
https://www.redhat.com/archives/libvirt-users/2014-November/msg00012.html

OpenStack generates it's own MACs so likely isn't affected by that side of things but in any case a more thorough check of the various code paths and implications seems prudent.

[Trent Lloyd (lathiat)]

Seems that viruuid.c:virUUIDGenerate tries to use virUUIDGenerateRandomBytes from /dev/urandom in preference to virUUIDGeneratePseudoRandomBytes which uses virRandomBits().

xenial version (1.3.1) does not contain the qemuDomainSecretAESSetup code. zesty (2.5.0) does though it seems to have a similar fallback.

So the most obvious paths may not be a security issue.

In terms of a proper fix... 2.5.0 also has a new virRandomBytes() that opens /dev/urandom to get bytes. Various code paths (as above) seem to use virRandomBits() as a fallback for urandom being unavailable -- so not sure if it makes more sense to attempt to seed virRandomBits with urandom anyway (but fall back) or just convert virMacAddrGenerate to use virRandomBytes. Particularly for a backport fix to trusty/xenial.

[Marc Deslauriers (mdeslaur)]

Hi Trent,

Could you please report this issue to the upstream developers here?

https://libvirt.org/securityprocess.html

Since you know all the details about the issue, you're in the best position to report it to them.

Thanks!

[Marc Deslauriers (mdeslaur)]

Hi Trent,

Did you get a chance to report this upstream yet?

[Trent Lloyd (lathiat)]

I identified that most likely there was minimal security impact, however I reported the issue upstream via the security contact anyway. They generally agreed exploit-ability didn't seem likely and so have simply applied a patch to stop using the bad RNG.

https://www.redhat.com/archives/libvirt-users/2018-May/msg00097.html
https://www.redhat.com/archives/libvirt-users/2018-May/msg00100.html

Setting this bug public and updating it with an SRU request to apply the upstream applied patch.