Comment 12 for bug 1858495

None of the VMs will be using spinning disks, it's all SSD; and virtual disks anyhow.

I would not expect much timing difference on virtual hardware; there aren't real device or pci timing delays; though the kernel may wait for them; however, it should be consistent.

In terms of the things that caught my eye:

# HotSpot maximum delta between kernel messages: 2.00000
# [ 2.846188] localhost.localdomain kernel: AES CTR mode by8 optimization enabled
# [ 5.919313] localhost.localdomain kernel: raid6: avx2x4 gen() 21512 MB/s

# [ 5.159028] localhost.localdomain kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
# [ 8.385284] localhost.localdomain systemd[1]: systemd 237 running in

[ 1.788581] localhost.localdomain kernel: * Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
                                             * this clock source is slow. Consider trying other clock sources
[ 3.545974] localhost.localdomain kernel: Unstable clock detected, switching default tracing clock to "global"
                                             If you want to keep using the local clock, then add:
                                               "trace_clock=local"
                                             on the kernel command line

Now, I owe you a /proc/cmdline, I'll get to that.

I looked over the spreadsheet; and while they are consistent with in each type (no template, after templating) the total time in kernel is double in after-templating (according to systemd-analyze); so I'd like to understand why that's the case.

% grep "Startup finished.*kernel" bug-bionic-baseline-no*.debug/*/journal.log | cut -d" " -f 7-
Startup finished in 3.209s (kernel) + 49.305s (userspace) = 52.515s.
Startup finished in 3.355s (kernel) + 51.732s (userspace) = 55.088s.
Startup finished in 3.287s (kernel) + 51.747s (userspace) = 55.035s.
Startup finished in 3.129s (kernel) + 50.066s (userspace) = 53.195s.
Startup finished in 3.350s (kernel) + 50.682s (userspace) = 54.032s.
Startup finished in 3.355s (kernel) + 49.322s (userspace) = 52.678s.
Startup finished in 3.219s (kernel) + 51.124s (userspace) = 54.343s.
Startup finished in 3.128s (kernel) + 49.226s (userspace) = 52.354s.
Startup finished in 3.193s (kernel) + 53.197s (userspace) = 56.390s.
Startup finished in 3.118s (kernel) + 46.203s (userspace) = 49.322s.

vs

% grep "Startup finished.*kernel" bug-bionic-baseline-after*.debug/*/journal.log | cut -d" " -f 7-
Startup finished in 7.685s (kernel) + 32.463s (userspace) = 40.148s.
Startup finished in 7.041s (kernel) + 35.998s (userspace) = 43.040s.
Startup finished in 7.808s (kernel) + 35.444s (userspace) = 43.253s.
Startup finished in 7.206s (kernel) + 37.952s (userspace) = 45.159s.
Startup finished in 8.426s (kernel) + 36.976s (userspace) = 45.403s.
Startup finished in 6.731s (kernel) + 35.484s (userspace) = 42.216s.
Startup finished in 7.152s (kernel) + 32.664s (userspace) = 39.817s.
Startup finished in 7.429s (kernel) + 36.177s (userspace) = 43.606s.
Startup finished in 9.075s (kernel) + 32.494s (userspace) = 41.570s.
Startup finished in 7.281s (kernel) + 32.732s (userspace) = 40.013s.