poor performance of resize2fs on 12.04

[Scott Moser]

I just ran a simple single instance launch, and on raring i saw:
  cc_resizefs.py[DEBUG]: Resizing took 0.450 seconds
on precise:
  cc_resizefs.py[DEBUG]: resize took 6.509745121 seconds

Those are real time reports from /var/log/cloud-init.log for a resize
up to 10G root from the ~1.4G that the images have built in.

This is known-fixed in current versions (quantal and later).
The changes were done to upstream e2fsprogs under debian bug 663237.

Bug 978012 is a general request for a newer e2fsprogs, and this bug is a specific request for the performance improvements.

Related bugs:
 * bug 978012: Please SRU micro bug fix release of e2fsprogs 1.42.4-3ubuntu1
   (main) from Quantal (main)
 * debian bug 663237: Make resize2fs shrinking use much less CPU

[Scott Moser (smoser)]

I have a branch at lp:~smoser/ubuntu/precise/e2fsprogs/resize2fs_performance/ where I think I cherry picked the fixes, but haven't done much testing yet.

Attaching a cloud-config that can be hopefully used to test this change, by adding the repo and then doing the resize after update/upgrade has occurred.

[Matt Rae (mattrae)]

I'm seeing slow resizefs from cloud-init.log when booting from volume, where the volume is 100G ceph rbd. I'm seeing high IO on ceph during this time even though I'm booting one instance at a time with a 20 disk ceph cluster. Is resize2fs an expensive operation?

2013-09-26 08:15:17,092 - cc_resizefs.py[DEBUG]: resize took 4269.06579185 seconds

[Matt Rae (mattrae)]

noting I'm using the current precise cloud image:
http://cloud-images.ubuntu.com/precise/20130926/

I'll have to confirm the resize2fs version. If there is any other details that would be helpful let me know.

[Matt Rae (mattrae)]

I'll potentially try a raring image to see if there is any difference

[Scott Moser (smoser)]

On Sat, 28 Sep 2013, Matt Rae wrote:

> I'll potentially try a raring image to see if there is any difference

I suspect that simply getting resize2fs from raring would get you a 10x
(or better) performance increase.

[Matt Rae (mattrae)]

Thanks Scott, yeah we tried the raring image and saw a 200x improvement. We booted a precise and raring image to compare and precise took 200sec for the resize while raring took 1.2sec. We're doing all of our testing with a raring instance right now.

I don't have a urgent need for precise but it is quite a bit slower for the initial boot.

[Bryan Quigley (bryanquigley)]

My testing indicates this is fixed when using the linux kernel 3.3 image. Can anyone confirm that it's not an e2fsprogs issue?

[Dave Chiluk (chiluk)]

It almost cherry-picks, but it appears to be pulling in a number of other fixes as well.

I'm going to have to spend some time reviewing this.

[Dave Chiluk (chiluk)]

So this turned out to be a much bigger job than I think can be safely SRU'ed. I narrowed the list down to 25-30 fixes depending on how you do it, neither of which built at the end. Even if I spent the requisite amount of time to get this building that doesn't mean I would have successfully handled all corner cases. This is way to dangerous to backport this feature into precise in a subsystem that the majority of our users depend on to store their data.

So I'm moving the precise track to will not fix.

[Dave Chiluk (chiluk)]

Well apparently will not fix is not an option for me, so Opinion will have to do.