livecd-rootfs: .pyc files stripped from desktop ISO livefs, makes python startup slower

[Steve Langasek]

It seems we've had code in livecd-rootfs for roughly forever to strip .pyc files out of the squashfs included on desktop images. There is an override to skip this for server and cloud images, but we discovered this recently because the missing .pyc files on ubuntu-core images causes very long start-up times for python apps on some armhf systems.

So the question is, is this still a sensible optimization on Ubuntu Desktop images? It was done at a time when space was tight; that's much less of a concern now than it was.

If this does still make sense for Desktop, we should probably at least invert the logic to make this opt-in rather than opt-out so that new image types on ARM don't get caught out with the problem.

[Iain Lane (laney)]

willcooke pointed me to this bug.

Is it possible to have a build with version of livecd-rootfs which doesn't run this hook, so that we can see how much space we're talking about?

I think it's possible to do this in a PPA, but I'm not actually sure how to go about that.

[Iain Lane (laney)]

Also, ubiquity re-byte-compiles the files in the target. This is code that could go away if no flavours are going to be doing this, although it's not harmful to keep it there..

[Iain Lane (laney)]

Never mind #1, I've done it. File diff attached to show that it worked. There are some other changes too, so these numbers aren't completely accurate.

root@raleigh:/home/laney/temp/reallytemp# ls livecd.ubuntu.squashfs -l # new
-rw-r--r-- 1 laney laney 1673199616 Sep 29 16:44 livecd.ubuntu.squashfs
root@raleigh:/home/laney/temp/reallytemp# ls ubuntu/livecd.ubuntu.squashfs -l # old
-rw-r--r-- 1 laney laney 1654173696 Sep 29 08:56 ubuntu/livecd.ubuntu.squashfs

Which is about a 19 MB difference.

WDYT?

Should we make this call for flavours too?

[Steve Langasek (vorlon)]

On Thu, Sep 29, 2016 at 04:44:00PM -0000, Iain Lane wrote:
> root@raleigh:/home/laney/temp/reallytemp# ls livecd.ubuntu.squashfs -l # new
> -rw-r--r-- 1 laney laney 1673199616 Sep 29 16:44 livecd.ubuntu.squashfs
> root@raleigh:/home/laney/temp/reallytemp# ls ubuntu/livecd.ubuntu.squashfs -l # old
> -rw-r--r-- 1 laney laney 1654173696 Sep 29 08:56 ubuntu/livecd.ubuntu.squashfs

> Which is about a 19 MB difference.

> WDYT?

It seems fine to me, since Ubuntu Desktop is not near any size limits
currently.

> Should we make this call for flavours too?

They should have the option, and we shouldn't make such a change during
release freeze. But I think the evidence is strong that we should reverse
the default.

[Iain Lane (laney)]

On Thu, Sep 29, 2016 at 05:00:28PM -0000, Steve Langasek wrote:
> On Thu, Sep 29, 2016 at 04:44:00PM -0000, Iain Lane wrote:
> > root@raleigh:/home/laney/temp/reallytemp# ls livecd.ubuntu.squashfs -l # new
> > -rw-r--r-- 1 laney laney 1673199616 Sep 29 16:44 livecd.ubuntu.squashfs
> > root@raleigh:/home/laney/temp/reallytemp# ls ubuntu/livecd.ubuntu.squashfs -l # old
> > -rw-r--r-- 1 laney laney 1654173696 Sep 29 08:56 ubuntu/livecd.ubuntu.squashfs
>
> > Which is about a 19 MB difference.
>
> > WDYT?
>
> It seems fine to me, since Ubuntu Desktop is not near any size limits
> currently.

I think it's okay too.

>
> > Should we make this call for flavours too?
>
> They should have the option, and we shouldn't make such a change during
> release freeze. But I think the evidence is strong that we should reverse
> the default.

I don't really understand this bit. Are you saying that we should *not*
change this for Yakkety for Ubuntu Desktop?

What I propose to do (now or next cycle, as decided) is make .pyc
stripping opt-in.

If this cycle, then opt every flavour except Ubuntu Desktop and the ones
already excluded in and mail announcing the change to give flavours a
chance to switch too.

At the start of next cycle, opt every flavour out and give the chance to
opt opt back in. If there are no takers, then we can delete the code
altogether.

Cheers,

--
Iain Lane [ <email address hidden> ]
Debian Developer [ <email address hidden> ]
Ubuntu Developer [ <email address hidden> ]

[Steve Langasek (vorlon)]

On Fri, Sep 30, 2016 at 01:22:22PM -0000, Iain Lane wrote:
> > > Should we make this call for flavours too?

> > They should have the option, and we shouldn't make such a change during
> > release freeze. But I think the evidence is strong that we should reverse
> > the default.

> I don't really understand this bit. Are you saying that we should *not*
> change this for Yakkety for Ubuntu Desktop?

> What I propose to do (now or next cycle, as decided) is make .pyc
> stripping opt-in.

> If this cycle, then opt every flavour except Ubuntu Desktop and the ones
> already excluded in and mail announcing the change to give flavours a
> chance to switch too.

> At the start of next cycle, opt every flavour out and give the chance to
> opt opt back in. If there are no takers, then we can delete the code
> altogether.

Ok. I'm comfortable with the desktop team making this decision for their
own images post-beta. I just don't want us to be making a change that
impacts any other flavors post-beta without their buy-in.

So if we do it now, the process is:

 - invert the logic
 - compile a complete list of all flavors building livefses and add them to
   the opt-in list
 - send mail to ubuntu-release or something encouraging flavors to drop
   themselves from the opt-in list if they're comfortable with the change
 - wait forever for everybody to do this

If we wait until the opening of the cycle, the process can be:

 - invert the logic
 - add support for opt-in list but leave it empty by default
 - send mail to ubuntu-release letting flavors know about the change,
   encouraging them to leave everything as-is unless they have specific size
   problems

The latter seems preferable to me, but means we have to leave this bug open
for 2 more weeks :)

[Iain Lane (laney)]

Please review the merge proposal.

Draft email to -release.

---
Dear flavours,

For a long time, we've been stripping .pyc files from live images in order to save space. This is a speed-vs-space tradeoff, as in the absence of .pyc files, applications need to either perform the byte compilation on first startup or suffer a speed penalty.

Towards the end of the 16.10 cycle, we reviewed our position to see if it still made sense in an era where most of our images are less space constrained. On an Ubuntu Desktop live CD, the saving is approximately 19 MB. Unfortunately I don't have good numbers for the speed gain, but it is noticable. We consider this an acceptable cost for faster app startup, and so for Zesty we will invert this default and make .pyc stripping opt-in instead of opt-out.

This is an announcement to make flavours aware. If you wish to opt back in, please reply to this email or raise a merge proposal against livecd-rootfs.

Cheers,
---

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.437

---------------
livecd-rootfs (2.437) zesty; urgency=medium

  * live-build/auto/config: Don't remove .pyc files any more - we don't care
    *that* much about the small space gain these days, and would prefer the
    speed win. (LP: #1627190)

 -- Iain Lane <email address hidden> Fri, 04 Nov 2016 11:02:13 +0000