Ubuntu Server 14.04.1 as VirtualBox Guest disk performance drops with more than once CPU

[Markus Fischer]

I'm running VirtualBox 4.3.18 on OSX 10.9.5 .

I install Ubuntu Server 14.04 64bit and performed this basic disk write test:

    dd if=/dev/zero of=bigfile bs=1024 count=5000000

Setup of host (MacBook Pro 2014):

- 16GB RAM
- 500GB SSD

The setup of the guest machine:

- 2GB RAM
- 60GB VDMK disk (only occupies 1-2 GB after minimal installation)

The result with 14.04 / 1 CPU: 13.6s / 376 MB /s
The result with 14.04 / 2 CPU: 41.5s / 123 MB /s

On 12.04 64bit with either 1 or 2 CPUs doesn't change the speed from around 376 MB/s

I came up with this simple `dd` test after noticing that just many things felt slow compared to my previous setup with 12.04. Running puppet, database migration, etc. Everything noticeable degraded after I ran a 14.04 with 2 CPUs.

The setup is made with the official 14.04 amd64 image and the standard installation procedure (just saying yes to all dialogs), no fancy/custom settings.

Things I tried:

- use different Motherboard chipset
- use IDE instead of SCSI (already during installation)
- use different IDE chipsets

I also created vmstat/iostat comparison (although I know about this tool but can't make anything up with it's output):

Before writing the 5GB zeroes, I startet vmstat/iostat and quit it afterwards; these are the logs. Maybe they help someone who understands them?

vmstat with 1 CPU

    procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
     r b swpd free buff cache si so bi bo in cs us sy id wa st
     0 0 0 1939068 16684 45944 0 0 2887 11 207 1421 5 9 85 0 0
     1 0 0 1834700 16844 147360 0 0 160 0 136 143 5 21 73 1 0
     0 1 0 1494508 16880 477948 0 0 32 196640 1742 215 17 80 0 3 0
     1 0 0 1216688 16904 748316 0 0 12 336720 3065 357 13 83 0 4 0
     1 1 0 916324 16912 1040380 0 0 8 300164 2490 275 15 85 0 0 0
     1 1 0 623784 16924 1325020 0 0 12 290818 2590 304 14 86 0 0 0
     1 0 0 331540 16932 1609576 0 0 8 277380 2602 261 15 84 0 1 0
     1 0 92 84544 5816 1860588 0 92 8 266408 2373 275 14 85 0 1 0
     3 0 416 69840 96 1880244 0 324 8 326136 2865 493 16 84 0 0 0
     1 0 724 81064 88 1869532 0 308 12 366776 3180 556 16 84 0 0 0
     1 0 1052 69352 92 1881392 0 328 12 336204 3074 541 17 82 0 1 0
     1 1 1480 80160 96 1870952 0 428 16 345776 2909 506 16 84 0 0 0
     1 0 1936 78492 104 1873396 0 456 12 347420 3146 547 16 81 0 3 0
     1 0 2544 67684 96 1884676 0 608 8 320156 2890 481 17 83 0 0 0
     3 1 3108 80780 96 1871980 0 564 12 350136 2974 507 16 83 0 1 0
     2 0 3548 79064 84 1873748 0 440 12 355388 3107 570 16 84 0 0 0
     1 0 3988 72260 96 1881724 0 440 16 319932 2990 505 17 83 0 0 0
     0 0 4112 77012 248 1877096 0 124 2712 269248 2306 350 4 43 38 14 0
     0 0 3980 75284 256 1878256 608 0 1748 76 63 110 0 2 97 1 0

### vmstat with 2 CPUs

    procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
     r b swpd free buff cache si so bi bo in cs us sy id wa st
     0 0 4728 72916 772 1879116 1 8 166 17813 252 111 7 13 79 1 0
     1 0 4728 1965416 852 37852 0 0 80 0 313 246 6 19 74 1 0
     1 0 4728 1838368 852 161568 0 0 0 0 494 21 21 29 50 0 0
     1 0 4728 1726976 856 269924 0 0 4 106516 1388 237 19 32 44 6 0
     1 0 4728 1604976 860 388704 0 0 4 106500 1357 189 22 36 40 3 0
     1 0 4728 1487140 880 503360 0 0 8 229588 2272 213 21 43 27 9 0
     1 0 4728 1357332 880 629844 0 0 0 0 506 33 22 29 50 0 0
     1 1 4728 1234228 884 749264 0 0 4 137664 1429 103 22 37 39 3 0
     1 0 4728 1112300 888 867996 0 0 4 128772 1629 123 21 37 38 4 0
     1 0 4728 987976 892 989032 0 0 4 106500 1329 83 22 35 41 3 0
     1 0 4728 871432 912 1102492 0 0 12 229424 2270 246 21 43 27 9 0
     1 0 4728 743556 912 1226964 0 0 0 0 486 18 22 29 50 0 0
     2 0 4728 620096 916 1346932 0 0 4 108188 1338 98 21 36 40 3 0
     1 0 4728 498700 920 1465012 0 0 4 158084 1737 145 22 39 35 4 0
     1 0 4728 375956 924 1584600 0 0 4 106500 1335 92 22 36 40 3 0
     1 0 4728 258288 940 1698692 0 0 8 225324 2223 210 21 43 27 9 0
     1 0 4728 132520 940 1821260 0 0 0 0 501 20 22 29 50 0 0
     1 0 4996 66768 112 1885580 0 268 220 106788 1511 219 19 34 41 5 0
     1 2 5012 71568 116 1880736 0 16 8 134292 1479 314 22 37 38 3 0
     1 0 5052 78080 116 1874516 0 40 4 132012 1719 275 21 40 36 4 0
     1 0 5088 67908 120 1884508 0 36 4 225356 2301 361 22 44 26 8 0
    procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
     r b swpd free buff cache si so bi bo in cs us sy id wa st
     1 0 5132 74848 112 1877920 0 44 0 44 589 263 22 29 49 0 0
     1 0 5172 72924 112 1879676 0 40 4 106556 1385 249 21 37 40 3 0
     1 1 5240 78988 116 1873796 0 68 4 137672 1476 277 22 37 38 3 0
     1 0 5308 70256 116 1882532 0 68 4 128712 1699 257 22 39 36 4 0
     1 0 5340 77468 136 1875524 0 32 16 225360 2289 366 21 44 27 8 0
     2 0 5388 77840 128 1875324 0 48 0 48 580 242 22 30 49 0 0
     2 0 5440 69500 120 1883484 0 52 4 106576 1387 248 22 37 39 3 0
     2 0 5496 76928 124 1876008 0 56 4 117564 1407 275 22 36 40 3 0
     1 0 5552 68904 124 1883932 0 56 4 148796 1754 281 21 40 35 4 0
     1 0 5600 70356 136 1882664 0 48 8 225372 2320 381 22 44 26 8 0
     1 0 5668 69292 132 1883832 0 68 0 68 572 241 22 30 49 0 0
     1 0 5720 74508 132 1878824 0 52 0 106572 1393 258 21 37 39 3 0
     1 0 5768 67348 120 1885924 0 48 4 106548 1494 272 22 37 39 3 0
     1 1 5836 73112 120 1880196 0 68 4 145992 1566 299 21 38 37 4 0
     1 0 5892 68260 132 1885004 0 56 8 239200 2562 400 22 46 24 9 0
     1 0 5976 66812 128 1886448 0 84 0 84 588 246 21 30 49 0 0
     1 0 6068 74444 132 1878868 0 92 4 106608 1407 316 22 36 39 3 0
     1 0 6140 65820 124 1887572 0 72 4 106572 1389 253 22 37 40 3 0
     1 1 6220 64992 124 1888248 0 80 4 137556 1506 288 22 37 38 3 0
     1 0 6296 69388 140 1884176 0 76 12 247672 2625 421 21 46 23 9 0
     1 0 6400 73392 124 1880312 0 104 0 104 599 237 22 30 49 0 0
    procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
     r b swpd free buff cache si so bi bo in cs us sy id wa st
     1 0 6484 71628 124 1882636 0 84 656 106604 1419 276 22 37 39 3 0
     0 1 6556 79784 120 1873452 0 72 8 221772 2087 355 17 37 39 7 0
     0 0 6556 78296 120 1876500 0 0 3036 39232 574 195 0 5 93 2 0

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1390354

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Markus Fischer (markus-fischer)]

When I run the command it opens a terminal browser but I can't managed to log in.

[Markus Fischer (markus-fischer)]

I found the problem:

The stock Ubuntu Server 14.04.1 amd64 iso image currently downloadable comes with a kernel which has the flag CONFIG_CONTEXT_TRACKING_FORCE=y . Recompiling the kernel without this flag set or upgrading to a kernel without the flag fixes the performance issues:

Multiple CPU, stock 14.04.1 kernel (3.13.0-32):

vagrant@ubuntu-14:~$ dd if=/dev/zero of=bigfile bs=1024 count=5000000
5000000+0 records in
5000000+0 records out
5120000000 bytes (5.1 GB) copied, 46.6511 s, 110 MB/s

Multiple CPUs, recompiled kernel (3.13.0-39 ; but I tested this specific kernel with both flag enabled/disabled):

vagrant@ubuntu-14:~$ dd if=/dev/zero of=bigfile bs=1024 count=5000000
5000000+0 records in
5000000+0 records out
5120000000 bytes (5.1 GB) copied, 10.4548 s, 490 MB/s

It seems this specific flag has been brought up in another report ( https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1349028 ) and is already fixed/soon to be released.