Big performance difference between debian and ubuntu guest

[Bryan Quigley]

Ubuntu trusty on trusty is substantially slower in apache benchmarks than debian on trusty. You can use PTS to see the difference here: http://openbenchmarking.org/result/1405191-PL-1405196PL73
I tested several releases and it does appear to be an Ubuntu vs Debian issue as opposed to a specific version. It also appears to be true if Debian is the host instead. The hosts have all been virtualized themselves (EC2, Virtualbox, kvm)

0. Started with m1.large image on canonistack (can give access if internal ip:10.55.60.136)
1. Install lxc and create an ubuntu trusty and debian jessie images, start them both.
 a. sudo lxc-create -t download -n ubuntu-trusty -- --dist ubuntu --release trusty --arch amd64
 b. sudo lxc-create -t download -n debian-jessie -- --dist debian --release jessie --arch amd64
2. Download and install PTS on both machines (I used lxc-attach...)
apt-get update; apt-get install wget; wget http://phoronix-test-suite.com/releases/repo/pts.debian/files/phoronix-test-suite_5.0.1_all.deb; dpkg -i phoronix-test-suite_5.0.1_all.deb; apt-get -f install
3. On both phoronix-test-suite install apache.
4. Then one at a time have one run: phoronix-test-suite benchmark apache and choose to upload the results. When it gives you the compare options, run that on the other node, phoronix-test-suite benchmark <insert the compare code after .../results/>

My results when doing the above, fresh: http://openbenchmarking.org/result/1405209-PL-1405200PL71

We are seeing a similar issue with another apache workload (not public).

[Bryan Quigley (bryanquigley)]

Very small performance loss when installing udev on debian (http://openbenchmarking.org/result/1405208-PL-1405209PL62). May not be significant.

My attempt to remove udev from Ubuntu did not succeed...

[Stefan Bader (smb)]

I would say that this points more likely to apache itself (or maybe some plumbing user-space parts). With that setup both containers use the same kernel and lxc parts. Only the user-space is from the given distro. I would not think udev is having much impact as that only runs when devices change (hot plugging disks or network devices for example). I don't think there is a lot of change there during the benchmark. I would probably look at how apache gets compiled on both distros (compiler flags or maybe security options being turned on in one but not the other)...

Oh, btw, about the ping yesterday, I think I should point out that I am Stefan and not Stéphane... ;)

[Bryan Quigley (bryanquigley)]

There definitely is a drop in performance from Ubuntu vs Debian normally.. see Virtualbox results: http://openbenchmarking.org/result/1405217-SO-1405214PL68

However, what can explain the drop in Ubuntu Trusty Native vs Ubuntu Trusty on LXC?

>Oh, btw, about the ping yesterday, I think I should point out that I am Stefan and not Stéphane... ;)
Sorry!

[Stefan Bader (smb)]

I assume that apache test is running against the local network interface, so we can rule out the additional network layers you get for lxc. Not sure here, Stéphane and Serge can probably come up with details, but all processes running inside the container probably belong to the same task group (as they get started from one common process). So scheduling could well be impacted when compared to bare-metal.

[Serge Hallyn (serge-hallyn)]

I installed the jessie apache2 packages (locally built) on trusty, and ran the benchmarks using first it and then with a (hand-built) trusty apache package. The results are at http://openbenchmarking.org/result/1405218-PL-TRUSTYHOS17 . The difference there is within SE. It could still be something in the toolchain. Currently setting up containers on that same host for more testing

[Bryan Quigley (bryanquigley)]

I'm not sure if PTS actually uses the system installed apache for it's tests.

[Serge Hallyn (serge-hallyn)]

My results in unprivileged containers on my own host:

http://openbenchmarking.org/result/1405216-PL-HETZNERUN74
http://openbenchmarking.org/result/1405216-PL-HETZNERUN44

[Serge Hallyn (serge-hallyn)]

Note that while I marked it confirmed, I don't see this as a bug in lxc.

Since we get the same results on bare metal, I would expect it to be the case in containers as well, barring drastic differences in kernel configurations.

I do wonder how much of this could be due to toolchain hardening flags when building the libraries.

In comment #3 you mention trusty in lxc vs trusty on bare metal. That's a separate issue, please open a new bug if you'd like to look into that.

[Serge Hallyn (serge-hallyn)]

Quoting Bryan Quigley (<email address hidden>):
> I'm not sure if PTS actually uses the system installed apache for it's
> tests.

Indeed looking more closely at the install logs suggests it builds its
own. You can find it in the jessie install under
./root/.phoronix-test-suite/installed-tests/pts/apache-1.6.1/httpd_

I wonder if we copy all of .phoronix-test-suite from one container to
the other and run it, what the results will look like.

lmbench results in both containers should also be interesting. If you
have a chance please do get and upload those.

[Bryan Quigley (bryanquigley)]

>I wonder if we copy all of .phoronix-test-suite from one container to
the other and run it, what the results will look like.

Indeed, that's the best way to confirm this. I'll give that a try. Thanks!

[Bryan Quigley (bryanquigley)]

Copying failed on missing/different libs.. I'm going to try to recreate the tests using the ab command/locally installed apache.

[Bryan Quigley (bryanquigley)]

Basically it looks like Debian in LXC might be faster looping back to itself, but once you do the test even from the host machine, they even out. Also results from Ubuntu apache installed on Debian Jessie.

I'm going to check to see where the other tests were run from.

Summary Results: http://pastebin.ubuntu.com/7498927/
Raw Results: http://pastebin.ubuntu.com/7498935/

[Bryan Quigley (bryanquigley)]

Thanks for the help. I'm going to close this per discussion above. I may end up opening a new bug (as per c#8) on trusty in LXC vs trusty on bare medal depending on the results of further testing.