dmidecode does not byte-swap for SMBIOS >= 2.6

[Peter Jose De Sousa]

Hi,

On VMWare 6.5 and higher (HW version 13 and higher) when decoding /sys/class/dmi/id/product_serial dmidecode produces an output whereby the first 16 characters have been reversed in comparison to /sys/class/dmi/id/product_uuid.

This manifests itself by breaking provisioning in Kubernetes and could cause other issues down the line. Other OS such as redhat have already patched this in their releases.

A demonstration of the issue can be found in this pastebin:

https://pastebin.canonical.com/p/qRfrpHXwGs/

VMWare Issue: https://kb.vmware.com/s/article/53609

Cheers!

[Tim Van Steenburgh (tvansteenburgh)]

Hi Peter, can you elaborate on this part please: "manifests itself by breaking provisioning in Kubernetes."

Can you describe the behavior you're seeing?

[Tim Van Steenburgh (tvansteenburgh)]

We could potentially patch Charmed Kubernetes against this by not reading /sys/class/dmi/id/product_uuid (https://github.com/charmed-kubernetes/charm-kubernetes-master/blob/master/reactive/kubernetes_master.py#L2244), but instead using the workaround published by vmware at https://kb.vmware.com/s/article/53609:

# dd if=/sys/firmware/dmi/entries/1-0/raw bs=1 count=24 status=none | python -c 'import struct; import sys; print("UUID: %08X-%04X-%04X-%04X-%04X%08X" % struct.unpack(">8xLHHHHL", sys.stdin.read()))'
UUID: 44454C4C-5200-1058-8053-B4C04F573132

[Peter Jose De Sousa (pjds)]

What we're seeing is a failed provisioning error when adding volumes, we hit this closed issue:

https://github.com/kubernetes/kubernetes/issues/58927 Kubernetes behaviour changed to fix this issue by switching to the */product_serial file as the source of truth.

But for our case the product_serial was incorrect, and the product_uuid was the source of truth, this meant when Kubernetes went to get the node it failed to find the VM:
https://github.com/kubernetes/kubernetes/blob/103e926604de6f79161b78af3e792d0ed282bc06/staging/src/k8s.io/legacy-cloud-providers/vsphere/vsphere_util.go#L656

[Pedro Guimarães (pguimaraes)]

Hi Tim, I don't think patching kubernetes-master will resolve this issue. Mainly because the issue happens on kubernetes-workers.

We were seeing PVCs failing with: "No VM found". When we look into the logs, actually kubernetes was trying to learn about its kubernetes-workers through SystemUUID.

Essentially:
https://github.com/kubernetes/kubernetes/blob/103e926604de6f79161b78af3e792d0ed282bc06/staging/src/k8s.io/legacy-cloud-providers/vsphere/vsphere_util.go#L656

Recovers UUID from SystemUUID, which according to https://github.com/kubernetes/kubernetes/issues/58927
Comes from product_serial

However, because of this dmidecode issue, the first half of the bytes are inversed (big endian vs. small endian).
What we saw on kubernetes-controller-manager log then was k8s grabbing a half-wrong UUID and trying to fetch it using FindVMUUID method. Of course that broke, since product_uuid held the right values.

To fix that, we have two ways:
1) work-around: run VMs with compatibility set for ESXi 5.5 and later; that forces vsphere to setup those VMs with older HW version (10) and there is no issue with dmidecode
2) Find out why dmidecode diverges when going to version 13 and fix it

[Tim Van Steenburgh (tvansteenburgh)]

Peter, Pedro, thanks for the extra detail. After reading https://github.com/kubernetes/kubernetes/issues/58927, I agree that we won't be able to fix this in Charmed Kubernetes directly.

[Peter Jose De Sousa (pjds)]

This has been fixed upstream in release 3.2[1] of dmidecode. So I guess we just need a backport?

Bionic currently has 3.1[2]

[1]: https://github.com/mirror/dmidecode/commit/7a9fc6d3cba86ec9c517b8ec314e099c5cb433b9
[2]: https://packages.ubuntu.com/bionic/dmidecode

[Peter Jose De Sousa (pjds)]

This bug seems like it could be related to: https://bugs.launchpad.net/ubuntu/+source/dmidecode/+bug/1236233

[Christian Ehrhardt  (paelzer)]

FYI:
19.04:
ii dmidecode 3.2-2 amd64 SMBIOS/DMI table decoder
$ sudo cat /sys/class/dmi/id/product_serial
VMware-42 18 59 50 7a fe ee b9-86 41 a6 c8 5a 5a 23 a6
$ sudo dmidecode | grep -i serial | grep VM
        Serial Number: VMware-42 18 59 50 7a fe ee b9-86 41 a6 c8 5a 5a 23 a6

Bionic:
ii dmidecode 3.1-1 amd64 SMBIOS/DMI table decoder
$ sudo cat /sys/class/dmi/id/product_serial
VMware-42 18 ac 2f fd b1 53 1b-8d ba ba ce ed cd d5 4b
$ sudo dmidecode | grep -i serial | grep VM
        Serial Number: VMware-42 18 ac 2f fd b1 53 1b-8d ba ba ce ed cd d5 4b

I can not see the issue over here at least

[Sergio Durigan Junior (sergiodj)]

Hi Pedro,

As Christian mentioned above, we tried to reproduce the issue on bionic but were not able to.

Interestingly, when I was looking at the package we ship on bionic, I noticed that the patch you mentioned on comment #6 is already included on dmidecode 3.1. Actually, if you look at the upstream repository, you can verify that the patch has been there since version 2.10 (from 2008).

Could you please provide a bit more information on how to reproduce the failure? Thanks!

[Peter Jose De Sousa (pjds)]

Hey both,

Was this experiment run with VMWare version 11 and higher?

Also /sys/class/dmi/id/product_uuid and /sys/class/dmi/id/product_uuid should not match up if the problem is being exhibited.

Cheers

[Sergio Durigan Junior (sergiodj)]

Hey Peter,

I'm trying to obtain access to the pool of VMs that Christian used to test this. I'll let you know when I have more info.

Thanks.

[Sergio Durigan Junior (sergiodj)]

OK, I finally got access to the VMs and was able to check the VMWare version there:

$ vmware-toolbox-cmd -v
10.3.0.5330 (build-8931395)

So it's pre-VMWare 11. And I double checked and confirmed that we can't reproduce the problem reported here indeed:

# apt policy dmidecode
dmidecode:
  Installed: 3.1-1
...
# cat /sys/class/dmi/id/product_uuid
42187693-2BD2-5AF3-BDA4-AAF4E13CA373
# cat /sys/class/dmi/id/product_uuid
42187693-2BD2-5AF3-BDA4-AAF4E13CA373

As I said above, the patch pointed at [1] is already present on 3.1-1 (in fact, it's been available since version 2.10), so there's something else going on here.

Were you able to confirm that dmidecode 3.2 fixes the issue for you?

Cheers.

[1]: https://github.com/mirror/dmidecode/commit/7a9fc6d3cba86ec9c517b8ec314e099c5cb433b9

[Peter Jose De Sousa (pjds)]

Hey Sergio

Just spunup a test machine on VMWare with version 13.

I can confirm the issue is still present:

pjds@sombrero:~$ sudo cat /sys/class/dmi/id/product_uuid
[sudo] password for pjds:
EB7F3642-93FE-BACA-6161-311E52960EBE
pjds@sombrero:~$
pjds@sombrero:~$ sudo cat /sys/class/dmi/id/product_serial
VMware-42 36 7f eb fe 93 ca ba-61 61 31 1e 52 96 0e be

Apologies, just realised I had a typo in my last message.

As you can see product_serial is not by swapped correctly thus not matching product_uuid.

Thanks for your help
Peter

[Peter Jose De Sousa (pjds)]

I should add that the experiment was run on Bionic.

I did however run this on Eoan and wasn't able to replicate the issue.

Thanks again!