Hash mismatch on "apt update"

[Andrew]

This is a really weird bug that is happening on Ubuntu 20.04 LTS (Live ISO!!!) and Kali 2020.2, but not Debian 10 (so, it affects at least apt 2.0.2ubuntu0.1 and does not affect 1.8.2.1). It also only occurs on a single PC (as far as I know). All testing was done in Virtualbox and moving VM's to another PC fixed issue (without changing anything inside the VM).

On running "apt update", there is an error "Hash Sum mismatch" which shows that SHA1 and SHA256 hashes differ from expected (while MD5 and file size is correct). E.g.:

  Hash Sum mismatch
  Hashes of expected file:
   - Filesize:314536 [weak]
   - SHA256:aa1c6c96b09a0c695dc475d99b407c675e564fbfe51b3e26230c6320b45666d0
   - SHA1:4f438d7e0c78dfb0486f86dc0a3dba30575eb617 [weak]
   - MD5Sum:5269212c54feb3dceabadb66583f6778 [weak]
  Hashes of received file:
   - SHA256:f47a968e7a10aff91df8b1d3f682ce11d161ff1b17056268b9ae1c10447523b2
   - SHA1:2839e062232ed234d0c04e60fe6b2a687c950e5b [weak]
   - MD5Sum:5269212c54feb3dceabadb66583f6778 [weak]
   - Filesize:314536 [weak]

I ran packet capture and extracted archives which are getting verified. All of their hashes are correct (exactly as expected).

It seems that calculating SHA1 and SHA256 the way APT does it produces wrong result, while running command line tools sha1sum and sha256sum (on the same PC inside the same VM) produces correct result.

I wrote the minimal reproducible example (hashtest.cc) that produces output such as this:

Calculating hashes same way apt does.

 - MD5Sum:c89b13b76197d0d554400e00e46c0740
 - SHA1:f6901a4486e69a1f503401daa02b520f1b0e22ba
 - SHA256:9075301b3961aca23b69bf2868a18dca184b383a0ec1de35516f0a8a182c2cb6
 - SHA512:7506f6f5c5d5e97f8c6ecac2489e7d6260002bd530370c6193a04620f94285dca0f5cf2bb9ead40afbd72fdf3a239349a57f81165b5b857af6ad7ddeab8da036
 - Checksum-FileSize:892549

Calculating hashes through command line tools.

 - md5sum: c89b13b76197d0d554400e00e46c0740
 - sha1sum: f6901a4486e69a1f503401daa02b520f1b0e22ba
 - sha256sum: 9075301b3961aca23b69bf2868a18dca184b383a0ec1de35516f0a8a182c2cb6
 - sha512sum: 7506f6f5c5d5e97f8c6ecac2489e7d6260002bd530370c6193a04620f94285dca0f5cf2bb9ead40afbd72fdf3a239349a57f81165b5b857af6ad7ddeab8da036

It's in the attachment alongside with an example file that causes this hash mismatch. There's also debug.log which contains various versions, etc (although as I said, it has been verified on latest Ubuntu Live ISO).

I have a suspicion that the bug is in the gcrypt library, not apt itself, but I haven't yet verified it. The libgcrypt20 version in Ubuntu is 1.8.5-5ubuntu1 (in Kali as well), while Debian 10 (which isn't affected) uses 1.8.4-5.
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu27
Architecture: amd64
CasperMD5CheckMismatches: ./casper/filesystem.squashfs
CasperMD5CheckResult: fail
CasperVersion: 1.445
DistroRelease: Ubuntu 20.04
LiveMediaBuild: Ubuntu 20.04 LTS "Focal Fossa" - Release amd64 (20200423)
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
Package: apt 2.0.2
PackageArchitecture: amd64
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 5.4.0-26.30-generic 5.4.30
Tags: focal
Uname: Linux 5.4.0-26-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True

[Chris Guiver (guiverc)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. Please execute the following command only once, as it will automatically gather debugging information, in a terminal:

apport-collect 1890006

When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

[Julian Andres Klode (juliank)]

Don't see any reason for any apport collecting here.

This needs a clean reproducer or someone with the issue to do the work to analyse what's wrong.

Looking at the bug report, the output from the reproducer matches that of the commandline tools?

[Andrew (0xf005ec)]

apport information

[Andrew (0xf005ec)]

apport information

[Andrew (0xf005ec)]

Here is apport stuff, and there are more news.

I confirmed that the problem was in the gcrypt. More precisely, disabling SSSE3 fixes this issue (echo intel-ssse3 > /etc/gcrypt/hwf.deny).

The Virtualbox (running on Windows 10) was using software virtualization ("execution engine: native API") instead of VT-x - this might be the root cause (i.e. bug is in Virtualbox), but I'm helping a friend and don't have direct access to that PC, so can't test what's blocking VT-x. Hyper-V and WSL are disabled in Windows features, the person hasn't checked whether VT-x is enabled in BIOS yet.

[Andrew (0xf005ec)]

VT-x status in BIOS is a red herring. Turns out modern Virtualbox won't even start VM's without VT-x.

A true cause turned out to be "Windows Sandbox" feature (although I imagine for some other users the problem might be triggered by "Windows Subsystem for Linux" or anti-virus software).

Here is how to reproduce it in Windows 10:

  1. Enable "Windows Sandbox" (Control panel -> Programs and Features -> Turn Windows features on or off)
  2. After restart run Ubuntu live ISO in virtualbox and notice that
    a) VM slows to a crawl, some apps (firefox) start crashing randomly, etc
    b) Virtualbox shows "green turtle" icon (third from the right on the lower panel) which lists Execution engine as "native API" instead of "VT-x"
    c) Running "apt update" produces error mentioning "hash mismatch"
    d) Running workaround (mkdir -p /etc/gcrypt; echo intel-ssse3 > /etc/gcrypt/hwf.deny) fixes the apt error
  3. Disable "Windows Sandbox" feature
  4. After Windows restart run Ubuntu in Virtualbox and be surprised by green turtle still being there, all mentioned problems still present
  5. Restart Windows second time, run Ubuntu in Virtualbox again, this time there is a blue chip instead of green turtle and apt works without workarounds

Clearly it seems that gcrypt isnt at fault here and the true culprits are Windows Sandbox (possibly some other mechanism that Windows Sandbox enables) and/or Virtualbox. However the result is that new users are getting wrong impression about GNU/Linux and Ubuntu in particular (it appears slow and lags during "live" phase, fails to update after install).

So I'd say Ubuntu should detect that it's being run in this broken virtualization mode, show the informative explanation to user (that their current experience isn't representative && how they can fix that) or at least enable the workaround for gcrypt (although if SSSE3 is buggy in this mode, I imagine more things are stealthily broken).

Surprisingly Ubuntu still finishes installation, even with default "update from internet" option enabled. After installation, "apt update" still produces the same "hash mismatch" errors though. Enabling workaround and rerunning "apt update" predictably finds many more (292 right now) packages that are to be upgraded. This fail-open behavior (installer and updater ignoring the errors) will no doubt lead to subtle bug down the road for the users as well (I'd argue that it's a security issue as well - users not being informed about security patches they're missing).

[Bálint (szybalint)]

I've spent a day trying to figure out why my VMs were not working until I found this report.

TLDR: Windows Sandbox in itself does not break the virtualization for Virtualbox, what does is the Virtual Machine Platform (on which I guess Windows Sandbox depends on).

I'll try to gather everything I experienced so someone smarter can make use of it:

So I upgraded to WSL2 which requires Virtual Machine Platform to be enabled. Couple of days later I tried opening an existing, already configured and previously working VM with Ubuntu Server 20.04, it got stuck at trying to perform some RAID6 xor check (excuse my incompetence). Shut down the VM, restore to a stable snapshot, now it got stuck at "Loading essential drivers...". Shut down, restore snapshot, reset voltages in Throttlestop (undervolt), reboot. Now the VM starts for some reason, super slow though, and I'm getting Hash Sum mismatch on apt update/upgrade. Tried a desktop Ubuntu VM, starts super slow, same mismatch error. WSL and my Linux server works fine on the same network, so that's not the issue. Finally I found this thread, and indeed, Virtualbox was running in native API execution engine, disabled Virtual Machine Platform in Windows, reboot, voila, works!

I guess don't upgrade to WSL2 if you intend to use Virtualbox (WSL1 does not need Virtual Machine Platform).

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Gwendal (mygg29)]

I am also affected by the bug,
echo all > /etc/gcrypt/hwf.deny fixed it.

The sha1 and md5 sum were equals but the sha256 sums were different.