[ubuntu-cpc] please make /tmp a tmpfs in RAM

[Dustin Kirkland ]

In Ubuntu, we have always cleared /tmp on every boot.

As such, on servers, by default /tmp should actually be a tmpfs entirely in RAM, when there is enough memory in the system. This threshold should be configurable by the end user (in cloud-init?), and default threshold of ~3GB.

Read about tmpfs here: https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt

This has several advantages, mainly:
 * Performance - much faster read/write access to data in /tmp
   - especially if your disk is spinning media
   - and if you're on SSD, this feature extends the life of your flash by reducing your NAND flash writes
 * Security - sensitive data would be cleared from memory on boot, rather than written (leaked) to disk -- important for encryption scenarios
 * Power consumption - storing information in memory is more energy efficient than reading and writing to disk

In scenarios where more space in /tmp is needed than available, one can compliment that tmpfs with 'sudo apt-get install swapspace' which will dynamically create/delete swapfile as necessary. See: http://manpg.es/swapspace

[Ben Howard (darkmuggle-deactivatedaccount)]

I am inclined to make the change here. The only problem that I can see is that some smaller cloud instance types (*.micro on AWS, the upcoming nano instance, GCE's micro, etc) this might be problematic.

[Ben Howard (darkmuggle-deactivatedaccount)]

As an alternative solution, I am wondering if we could make this be a cloud-init function to give users control, with a default of /tmp being a tmpfs when memory is sufficient (i.e. if you have less than 1G of RAM, /tmp is tmpfs or you are in a container).

The other area where I think that this change might create a problem is in high density container scenarios (running 200 LXC containers).

[Robie Basak (racb)]

> * Performance - much faster read/write access to data in /tmp

Is this really true? Writes to /tmp will go to the page cache, which I believe is an identical path whether /tmp is backed by disk or by tmpfs. Similarly reads from /tmp will come from the page cache except where pages have been evicted in the case of a disk-backed /tmp, which cannot happen with tmpfs.

fsyncs on /tmp will be slower. Whether that's a problem depends on the application. But do we need to use tmpfs to eliminate that? Is there a better way of just swallowing syncs (eatmydata style), which would have the same effect?

The big disadvantage of a tmpfs /tmp is that it cannot be paged out, and thus puts pressure on available system RAM. One failure case is a sysadmin expecting it to be backed to disk (and therefore be big), using it for something temporary, and then killing the system due to memory starvation.

> * Security - sensitive data would be cleared from memory on boot, rather than written (leaked) to disk -- important for encryption scenarios

If this is important then surely the user is encrypting the filesystem on disk anyway?

[Robie Basak (racb)]

> and then killing the system due to memory starvation

Actually I think tmpfs is limited to 50% of system RAM or something by default, but that brings up another issue. By doing this we're severely limiting the available disk in /tmp. Or does that not matter in the cloud image case because / is expected to be smaller than 50% of RAM anyway?

[Robie Basak (racb)]

I'd also add that the cloud image build process isn't the right place to make such a change, I don't think. It should be cloud-init or some system boot script that is shipped by a package in the archive.

[Robie Basak (racb)]

> The big disadvantage of a tmpfs /tmp is that it cannot be paged out

My mistake, this is untrue if you have swap enabled (and enough swap available). I was thinking of ramfs. I still wonder what any performance gain would be, though. Using tmpfs will still limit size to available RAM + available swap space.

[Ryan Harper (raharper)]

Can we publish some test images? Instead of guessing at this, we can benchmark this.

In general swapping ram-based for what is almost always disk-based is going to impact applications/deployments using tmp and expecting enough space there. It's not uncommon for large ISO or other image downloads to reside in this location and this change would break smaller instances which are memory constrained from doing these sorts of tasks without changing where things are stored.

[Moshe Katz (kohenkatz)]

This doesn't sound like a good idea to me, given the prevalence of running Ubuntu on virtual machines with less than 4GB of RAM. I'm thinking specifically of hosting providers like Linode, AWS, DigitalOcean, etc., as well as tools like Vagrant, all of which are extremely popular right now. Using tmpfs on a machine with so little RAM doesn't sound like it will really help performance because the gain in speed accessing /tmp is offset by having to swap to disk.

[Jarno Suni (jarnos)]

swapspace sounds cool. I'll try it on my next installation of Ubuntu instead of reserving a separate partition.

I think the Ubuntu installer could have an option for making /tmp a tmpfs and for making suitable swap configuration for that thus adapting to all needs. I use noatime option for filesystems that may have read operations on SSD to reduce wear, but that might not work in every setup. I think also notebooks benefit from such optimizations, as they tend to have more often only SSD as local mass storage.

[Jarno Suni (jarnos)]

If user has chosen encrypted home directory during installation of Ubuntu, he/she probably wants encrypted swap, too. I am not sure, if that is possible with swapspace.

[Ben Howard (darkmuggle-deactivatedaccount)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

After considering the robust discussion, I would like to propose that
we move forward with this change via Cloud-init with some sane
defaults (i.e. it is _NOT_ running in a container and memory is
greater than 2GB of RAM).

Given that Xenial is in Alpha, now is the time to make this change.
The reality is that without actually making the change we can't and
won't be able to fully scope the impact.

So with that, I think that having Cloud-init configure /tmp as tmpfs,
while giving the user the ability to change it, is a sane path
forward. In the event that we have serious or unintended consequences,
we can revert pre-release or even via an SRU if necessary.
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJWpm/mAAoJEETWil5UBqhmczQQAIX8/VSpF4IsVQsJT+YsItdO
QYy4GTYSDU8MI4wZIcs448epq31QBZyFCijRiJLPSD72I9izAAbbjzSYnUMEivCH
uCEHMFTkSZyQQU0x/2LHpsjMlcgQiM3izfjI6dRf0y8qE36oKTqSWR3lETwDl8ag
nzimb8JuGgFvwLMznFipH5lIJQlu75GGrAvhqql3vnEsY9B76dVxD00FPQvWkoOH
gC5F1HC/KpleuJbX1Fz3wIUu6fzYJbYedRbiCJ12ioeZdternx3Nk0/NzLcauzmr
lbFJhpNRSxKtLMGubwNEbJ/vDJC+aC8cJj1NGANe8g+T61xK/vmmgJUoERXZp+0N
P2I5gx9EvGARTePEaciU1E/i18Jv9t72nCvn4aeGNPDG2nAZsXL6Gygw98FtEqBg
3mP7IU6+5bMZw6VR+7oPXiW2+Auq1KRK4sxi0N5uRINq3VHbDI8l+K0Aewh7jCR1
1uC/I5CJWNXP3vSH6b1P2pBF92DQ0xkEeArJSw8jo4B9kacS85BVMNg2QlGoK0A4
ico/NvLMlFTJFtePYXxUBKjLuDDqw64k8U4poJxejwJVDQ/qmt2zz97jwfkaQBkZ
fdKx17tr7NsdRgxYG37gQrhD7KsxEolsEyUE3vn+69pf45i8CWRHQssJ+C0xxVLm
Bv19ys9hbgMspkSpWUp4
=YTxu
-----END PGP SIGNATURE-----

[Robie Basak (racb)]

Hi Ben,

On Mon, Jan 25, 2016 at 06:56:38PM -0000, Ben Howard wrote:
> After considering the robust discussion, I would like to propose that
> we move forward with this change via Cloud-init with some sane
> defaults (i.e. it is _NOT_ running in a container and memory is
> greater than 2GB of RAM).
>
> Given that Xenial is in Alpha, now is the time to make this change.
> The reality is that without actually making the change we can't and
> won't be able to fully scope the impact.
>
> So with that, I think that having Cloud-init configure /tmp as tmpfs,
> while giving the user the ability to change it, is a sane path
> forward. In the event that we have serious or unintended consequences,
> we can revert pre-release or even via an SRU if necessary.

I think it's quite clear that:

1) Ubuntu developers have not yet reached consensus on this (based on
the ubuntu-devel thread).

2) Some use cases *will* be impacted by this change (that is not to say
that we shouldn't do it, but I think we do need to consider the plight
of these use cases).

Given the time constraints, I suggest that we ask the tech board to make
a quick decision on this. Without consensus, my understanding of the
code of conduct says that this is what we should do. Who will drive
this?

I'd also ask that, before making the change, we have decent instructions
for 1) impacted users, for the non-default case; and 2) impacted
upstreams, so they know what they should do for the default case where
they know that a tmpfs /tmp won't do. Who can commit to figuring the
details out here?

I propose that there are only really two options to choose from
technically in terms of actually making the change:

1) Make the option available but non-default on cloud images.
2) Make the option available and default on cloud images (so opt-out).

Any more? I presume any option would be via cloud-init userdata?

Robie

[Dean Henrichsmeyer (dean)]

Let's not move forward with this right now. This decision needs more data and more consensus before being actioned.

[Ben Howard (darkmuggle-deactivatedaccount)]

Dropping this as won't fix.

[Jarno Suni (jarnos)]

As for encrypting swap created by swapspace, I have a question: http://askubuntu.com/questions/726577/how-can-you-setup-encrypted-swap-with-swapspace