Unmount of an fs with dirty cache buffers causes pathological slowdown

[Kees Cook]

I'm seeing huge latency sometimes when unmounting ext4 partitions. Steps that triggered this most recently:

sudo -s
cd /tmp
dd if=/dev/zero of=test.ext4 bs=1 count=1 seek=1G
mkfs.ext4 -F test.ext4
mkdir -p /mnt/test
mount -o loop test.ext4 /mnt/test
echo $(seq 65536) | (cd /mnt/test; xargs touch)
time umount /mnt/test

I'm seeing upwards of 5 minutes for the umount to finish.

[Kees Cook (kees)]

Additional note: my /tmp is on ext4, part of my root partition. (Which on on an LV, on MD)

[Jamie Strandboge (jdstrand)]

The reproducer works here. I use ext4 on MD (raid1) for '/' with /tmp as part of '/'. Additionally, I get the following trace in dmesg when trying to umount:
[214920.810112] INFO: task umount:16606 blocked for more than 120 seconds.
[214920.810118] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[214920.810121] umount D 00000000ffffffff 0 16606 16551 0x00000000
[214920.810128] ffff8800b27cbd18 0000000000000082 0000000000015b00 0000000000015b00
[214920.810135] ffff8800bcf15f80 ffff8800b27cbfd8 0000000000015b00 ffff8800bcf15bc0
[214920.810140] 0000000000015b00 ffff8800b27cbfd8 0000000000015b00 ffff8800bcf15f80
[214920.810146] Call Trace:
[214920.810158] [<ffffffff81162770>] ? bdi_sched_wait+0x0/0x20
[214920.810163] [<ffffffff8116277e>] bdi_sched_wait+0xe/0x20
[214920.810170] [<ffffffff815553ff>] __wait_on_bit+0x5f/0x90
[214920.810174] [<ffffffff81162770>] ? bdi_sched_wait+0x0/0x20
[214920.810181] [<ffffffff815554a8>] out_of_line_wait_on_bit+0x78/0x90
[214920.810187] [<ffffffff810835c0>] ? wake_bit_function+0x0/0x40
[214920.810192] [<ffffffff81162734>] ? bdi_queue_work+0xa4/0xe0
[214920.810196] [<ffffffff81163aaf>] bdi_sync_writeback+0x6f/0x80
[214920.810201] [<ffffffff81163ae0>] sync_inodes_sb+0x20/0x30
[214920.810205] [<ffffffff811675c2>] __sync_filesystem+0x82/0x90
[214920.810209] [<ffffffff811677cb>] sync_filesystem+0x4b/0x70
[214920.810214] [<ffffffff811422c7>] generic_shutdown_super+0x27/0x100
[214920.810218] [<ffffffff811423d1>] kill_block_super+0x31/0x50
[214920.810222] [<ffffffff811434a1>] deactivate_super+0x71/0x90
[214920.810227] [<ffffffff8115bc1f>] mntput_no_expire+0xbf/0x110
[214920.810231] [<ffffffff8115c357>] sys_umount+0x57/0xc0
[214920.810236] [<ffffffff810131f2>] system_call_fastpath+0x16/0x1b

[Jamie Strandboge (jdstrand)]

Also, it took more than 22 minutes to unmount here:

$ time umount /mnt/test

real 22m47.719s
user 0m0.000s
sys 0m0.180s

[Kees Cook (kees)]

This is getting worse the longer my system runs. Going to reboot...

$ time sync
real 103m5.920s

[Kees Cook (kees)]

This is reproducible after a reboot (9 minutes to umount). I'm seeing the dmesg warnings too.

[Phillip Susi (psusi)]

Seeing the same thing. Also using lvm, and call trace of umount while in D state looks the same.

[Phillip Susi (psusi)]

It seems like it only happens when there are plenty of dirty buffers that need flushed, then it hangs waiting for the sync to complete, even though that is done after maybe 30 seconds. If you wait a while for the system to settle, or sync before umount, then it is fine. It also may be related to the journal. When I disable the journal it does not seem to hang.

For that matter, there don't seem to be a bunch of dirty buffers left around when I don't use the journal. Each time I make a clean fs, and restore a dump to it, then unmount it. With the journal, the umount ( or sync if I do that first ) flushes a bunch of data to the disk, but without the journal it does not.

[Surbhi Palande (csurbhi)]

@Kees, Can you please do the following:

1) echo 1 >/proc/sys/kernel/lock_stat
fire the unmount command.
2) less /proc/lock_stat
If you could attach the output of the previous command then we could check if there is some lock contention occurring.

You could then disable lock_stat as follows:
3) echo 0 >/proc/sys/kernel/lock_stat
4) echo 0 > /proc/lock_stat

Thanks!

[Kees Cook (kees)]

# grep LOCK_STAT /boot/config-$(uname -r)
# CONFIG_LOCK_STAT is not set

[Kees Cook (kees)]

Might be upstream bug https://bugzilla.kernel.org/show_bug.cgi?id=14430

[Kees Cook (kees)]

@Surbhi: I can't use lock_stat because the stalled mount triggers lockdep warnings.

[Theodore Ts'o (tytso)]

I can confirm this using a bleeding edge kernel 2.6.34-rc2+, and when using a tmpfs mounted for /tmp, it takes about two minutes on a T400 laptop.

Using blktrace, it looks like we're doing a whole ton of journal writes after the umount.

Here's the blktrace summary of the reproducer:

Total (loop0):
 Reads Queued: 0, 0KiB Writes Queued: 74944, 299776KiB
 Read Dispatches: 0, 0KiB Write Dispatches: 0, 0KiB
 Reads Requeued: 0 Writes Requeued: 0
 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB
 Read Merges: 0, 0KiB Write Merges: 0, 0KiB
 IO unplugs: 66567 Timer unplugs: 0

And here's the blktrace of the reproducer adding a "sync" before the "time umount /mnt/test":

Total (loop1):
 Reads Queued: 2, 8KiB Writes Queued: 9438, 37752KiB
 Read Dispatches: 0, 0KiB Write Dispatches: 0, 0KiB
 Reads Requeued: 0 Writes Requeued: 0
 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB
 Read Merges: 0, 0KiB Write Merges: 0, 0KiB
 IO unplugs: 55 Timer unplugs: 0

Analyzing the blktrace logs more carefully, it looks like the umount path is doing a synchronous inode sync for each inode, which means that we're executing a journal transaction and a barrier between every single inode update. Doh!

I haven't analyzed the kernel code yet (probably won't have time tonight), but that does seem to be what's going on. Hopefully the kernel fix should be simple....

[Kees Cook (kees)]

I can confirm that add "sync" at the top of /etc/schroot/setup.d/10mount solves all the build stalls I've been seeing.

[Theodore Ts'o (tytso)]

The other place you'll want to drop a sync call is in whatever path gets used when someone unmounts a file system in Nautilus (or any other desktop file manager) by clicking on the "eject" icon. This is the one that is going to be more visible to users, since a common usage scenario is someone who drags a whole bunch of files to a USB disk or flash disk, and then clicks on the eject button --- and we don't want another repeat of the infamous "USB writes are slow: random performance bug" Launchpad debacle that goes on and on for years..... :-)

Thanks for turning this up, I'll try to get the root cause fixed and you guys can decide if you have time to backport a kernel fix to the Lucid kernel...

[Surbhi Palande (csurbhi)]

Kees Cook, I debugged this problem and found out that this is happening because of waiting on ext4 journals. I can see currently there is a ext4 thread going on for doing journal commit in two phases:

Phase 1) wake up the journal commit threads
(Do some other work)
Phase 2) wait for the commit to finish.

Once that is done we should have a solution. I will follow-up this thread and pick the solution from there.

[Surbhi Palande (csurbhi)]

There is a discussion on ext4 mailing list about improving the jbd2 layer (and not making journalling sync in two phases - however I will like to discuss this solution there.) I have also a few more findings...will try making a good solution out of it and get back here.

[Surbhi Palande (csurbhi)]

I did not read Teds comments above earlier :( !

Here is what i found out too:

1) barrier is called for every inode - but this happens for ext3 code too and ext3 unmount is not taking as much time.
2) journal transaction is also calling (one more) barrier for every inode.
2) putting timing information shows that sync does not wait as long on the barrier as unmount does - I am currently looking at why this is happening. After a sync is called when you call an unmount, a barrier is called on the same inodes for which sync already called a barrier.
Shall add more findings soon!

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 2.6.32-21.31

---------------
linux (2.6.32-21.31) lucid; urgency=low

  [ Andy Whitcroft ]

  * allow modules.builtin to be optional
  * d-i: add mpt2sas to the message-modules udeb
    - LP: #530361

  [ Christopher James Halse Rogers ]

  * SAUCE: Nouveau: Add quirk framework to disable acceleration
    - LP: #544088, #546393
  * SAUCE: Nouveau: Disable acceleration on MacBook Pros
    - LP: #546393
  * SAUCE: Nouveau: Disable acceleration on GeForce3 cards
    - LP: #544088
  * SAUCE: Nouveau: Disable acceleration on 6100 cards
    - LP: #542950

  [ Stefan Bader ]

  * SAUCE: dma-mapping: Remove WARN_ON in dma_free_coherent
    - LP: #458201

  [ Surbhi Palande ]

  * SAUCE: sync before umount to reduce time taken by ext4 umount
    - LP: #543617

  [ Upstream Kernel Changes ]

  * tipc: Fix oops on send prior to entering networked mode (v3)
    - CVE-2010-1187
  * KVM: x86 emulator: Add Virtual-8086 mode of emulation
    - LP: #561425
  * KVM: x86 emulator: fix memory access during x86 emulation
    - LP: #561425
  * KVM: x86 emulator: Check IOPL level during io instruction emulation
    - LP: #561425
  * KVM: x86 emulator: Fix popf emulation
    - LP: #561425
  * KVM: Fix segment descriptor loading
    - LP: #561425
  * KVM: VMX: Update instruction length on intercepted BP
    - LP: #561425
  * KVM: VMX: Use macros instead of hex value on cr0 initialization
    - LP: #561425
  * KVM: SVM: Reset cr0 properly on vcpu reset
    - LP: #561425
  * KVM: VMX: Disable unrestricted guest when EPT disabled
    - LP: #561425
  * KVM: x86: disable paravirt mmu reporting
    - LP: #561425
  * AppArmor: Fix put of unassigned ns if aa_unpack fails
  * AppArmor: Fix refcount bug when exec fails
    - LP: #562063
  * AppArmor: Take refcount on cxt->profile to ensure it remains a valid
    reference
    - LP: #367499
  * AppArmor: fix typo in scrubbing environment variable warning
    - LP: #562060
  * AppArmor: fix regression by setting default to mediate deleted files
    - LP: #562056
  * AppArmor: fix refcount order bug that can trigger during replacement
    - LP: #367499
  * AppArmor: Make sure to unmap aliases for vmalloced dfas before they are
    live
    - LP: #529288
  * AppArmor: address performance regression of replaced profile
    - LP: #549428
  * AppArmor: make the global side the correct type
    - LP: #562047
  * AppArmor: use the kernel shared workqueue to free vmalloc'ed dfas
  * sky2: add register definitions for new chips
    - LP: #537168
  * sky2: 88E8059 support
    - LP: #537168
  * net: Fix Yukon-2 Optima TCP offload setup
    - LP: #537168
  * net: Add missing TST_CFG_WRITE bits around sky2_pci_write
    - LP: #537168
  * sky2: print Optima chip name
    - LP: #537168
  * (Upstream) dell-laptop: defer dell_rfkill_update to worker thread
    - LP: #555261
  * drm/nv40: add LVDS table quirk for Dell Latitude D620
    - LP: #539730
 -- Andy Whitcroft <email address hidden> Tue, 13 Apr 2010 18:50:58 +0100

[Phillip Susi (psusi)]

On 4/13/2010 4:30 PM, Launchpad Bug Tracker wrote:
> * SAUCE: sync before umount to reduce time taken by ext4 umount
> - LP: #543617

This sounds more like a temporary workaround than a fix of the real bug.
 Is that the case and why? Just can't find the real problem, or it will
take too long to fix?

[Theodore Ts'o (tytso)]

On Tue, Apr 13, 2010 at 09:13:49PM -0000, Phillip Susi wrote:
> On 4/13/2010 4:30 PM, Launchpad Bug Tracker wrote:
> > * SAUCE: sync before umount to reduce time taken by ext4 umount
> > - LP: #543617
>
> This sounds more like a temporary workaround than a fix of the real bug.
> Is that the case and why? Just can't find the real problem, or it will
> take too long to fix?

I recommended doing a sync in userspace (i.e., in various shutdown
scripts and GNOME/KDE desktops) as a temporary workaround because I
didn't have time to poke at this before the Lucid release deadlines
(which is coming quite rapidly, yes). I guess the Ubuntu kernel team
decided it was easier drop a forced sync into the kernel. I haven't
examined the patch that they ultimately chose, but presumably it's low
risk to be inserted less than two weeks before the final release date
of Lucid if it was coded correctly. Me, I'd probably would have stuck
the sync in userspace, but I'm super paranoid this close to a
"enterprise-quality" release date, which is what the Lucid LTS release
purports to be.

As far as "trying to find the real problem", if Ubuntu was paying my
salary I'd give it more time to find the root cause of this bug, but
this is a low priority bug given other things on my plate. Red Hat
employs several very high powered file system developers, so they fix
a lot more of their own distro-specific bugs. Interestingly, this is
something that hasn't shown up as a complaint on Fedora systems. I'm
not sure why; the test case Kees provided shows that this is
definitely an upstream problem, but apparently something about their
choice of desktop components or how they are configured or something
about their init/hal scripts means that it's not showing up for their
users in practice for some reason.

My problem is I'm incredibly and busy at the moment, and I've already
done Ubuntu a huge favor by spending ten minutes to do a quickie
investigation. Ubuntu needs to learn that it can't rely on upstream
developers to jump through flaming hoops on short notice before a LTS
release deadline as a cost-saving mechanism to avoid hiring their own
senior kernel engineers. So hiring Surbhi is definitely a step in the
right direction. (One step on a journey of ten thousand, but a step
in the right direction nonetheless. :-)

Surbhi will eventually have the experience of folks like Eric Sandeen
and Josef Bacik, or Jan Kara at SuSE, and eventually hopefully she'll
be able to fix bugs like this quickly. Someone who is an ext4 expert
probably could localize this down in less than a day, especially given
my "ten minute investigation" to point them in the right direction.
The fact that "sync" on the command line causes the right thing to
happen, and "umount" with dirty inodes extant, doesn't, is a pretty
strong hint of where to look, and no, the root cause is probably not
the jbd2 layer as Surbhi has suggested.

      - Ted

P.S. Next thing for Ubuntu to learn --- how to pay their engineers
well enough, and how to give them enough time to work on upstream
issues, that once they gain that experience on Ubuntu's dime and
become well known in the open source community, they don't ...

[Kees (kees-redhat-bugs)]

Description of problem: umount on ext4 has seriously regressed, stalling for over a minute (or more):
https://bugzilla.kernel.org/show_bug.cgi?id=15906

Version-Release number of selected component (if applicable): 2.6.33

How reproducible: always

Steps to Reproduce:
1. install a Fedora 13, accepted defaults.
2. Run the following as root:

cd /tmp
dd if=/dev/zero of=test.ext4 bs=1 count=1 seek=1G
mkfs.ext4 -F test.ext4
mkdir -p /mnt/test
mount -o loop test.ext4 /mnt/test
echo $(seq 65536) | (cd /mnt/test; xargs touch)
time umount /mnt/test

Actual results: over a minute to perform umount.

Expected results: a few seconds.

[Kees Cook (kees)]

The sync()-in-kernel work-around only seems to help on an otherwise idle system. I've opened an upstream and Fedora bug now, since I've been able trivially reproduce it both on current Fedora and the latest stable upstream kernel.

[Eric (eric-redhat-bugs)]

Looks barrier related, if you change the mount -o loop to mount -o loop,nobarrier, it's speedy.

commit 68db1961bbf4e16c220ccec4a780e966bc1fece3
Author: Nikanth Karthikesan <email address hidden>
Date: Tue Mar 24 12:29:54 2009 +0100

    loop: support barrier writes

    Honour barrier requests in the loop back block device driver.
    In case of barrier bios, flush the backing file once before processing the
    barrier and once after to guarantee ordering. In case of filesystems that
    does not support fsync, barrier bios would be failed with -EOPNOTSUPP.

    Signed-off-by: Nikanth Karthikesan <email address hidden>
    Signed-off-by: Jens Axboe <email address hidden>

So you're fsyncing the loop file 2x for each barrier, and those fsyncs in turn cause cache flushes on your underlying device; once upon a time lvm didn't do anything there, but now it does ... I think you're just getting bitten by the new and improved barrier support at every level.

-Eric

[Eric (eric-redhat-bugs)]

Since this really seems to be an upstream-relevant bug motivated by an ubuntu bug that ubuntu seemed unable to solve, I'm going to close this as UPSTREAM and worry about it there, rather than working the ubuntu bug through the fedora bugzilla.

[Eric (eric-redhat-bugs)]

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/543617/comments/21

[Zooko Wilcox-O'Hearn (zooko)]

I wonder if this was the cause of the 25x slowdown of apache as reported by Phoronix: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/567302

[Michael (michael-redhat-bugs)]

The mad skillz that did the new colour schemes for a new release of an os (Ubuntu 10.04) should be just as capable at being able to overcome the limitation of journal transaction and barrier blocks between every single inode update by breakfast.

[Fedora (fedora-redhat-bugs)]

kernel-2.6.33.4-95.fc13 has been submitted as an update for Fedora 13.
http://admin.fedoraproject.org/updates/kernel-2.6.33.4-95.fc13

[Fedora (fedora-redhat-bugs)]

kernel-2.6.33.4-95.fc13 has been pushed to the Fedora 13 stable repository. If problems still persist, please make note of it in this bug report.

[Brian Bloniarz (brianbloniarz)]

The committed workaround for this syncs _all_ filesystems on _any_ umount(), if
I understand it correctly.

This has caused a regression for us; some of our scripts mount a tmpfs and
then umount it. Since that now syncs every filesystem, it can take much
longer than it should.

What is the status of a real fix for this bug? Now that the upstream bug has been
closed (?), will the workaround be going away any time soon.

[Brian Bloniarz (brianbloniarz)]

I've submitted an additional bug for the tmpfs regression, https://bugs.launchpad.net/ubuntu/+source/linux/+bug/585092

[Andy Whitcroft (apw)]

The fixes referred to in bug #585092 seem to have introduced regressions and are currently reverted.

[Steve Conklin (sconklin)]

I have test kernels built with two candidate patches, the kernels are here:
http://kernel.ubuntu.com/~sconklin/kernels-bug543617/

Please test and report.

The branch I built this from is here:
http://kernel.ubuntu.com/git?p=sconklin/ubuntu-lucid.git;a=shortlog;h=refs/heads/bug543617

I'll add the patches to this bug following this comment

[Steve Conklin (sconklin)]

Um, the test kernels don't have the bad patch reverted. I'll rebuild and get new ones up.

[Steve Conklin (sconklin)]

Patch 5e1941884c700b7b97bcad52c2d8212ac56a7ebc. must be reverted in addition to applying these two patches

[Steve Conklin (sconklin)]

ok, once more with feeling!

I have test kernels built with two candidate patches, the kernels are here:
http://kernel.ubuntu.com/~sconklin/kernels-bug543617/

This time the incorrect patch has been reverted also.

Please test and report.

The branch I built this from is here:
http://kernel.ubuntu.com/git?p=sconklin/ubuntu-lucid.git;a=shortlog;h=refs/heads/bug543617

Patches are in comments #27 and #28

[Max Bowsher (maxb)]

Steve:

Well, your kernel certainly avoided the slowness I was experiencing in unmounting bindmounts, though in fairness merely reverting the SAUCE would most likely have been all I needed.

Unfortunately your kernel somehow made compiz refuse to start, which is pretty weird.

[Stefan Bader (smb)]

A batch of backported patches from upstream and the revert of the SAUCE patch to minimize umount times has been committed to the repo. At the moment updated kernels will only be available from https://launchpad.net/~kernel-ppa/+archive/pre-proposed until the current kernel in proposed clears to updates.

[Steve Conklin (sconklin)]

SRU Justification

Impact: The upstream process for stable tree updates is quite similar in scope to the Ubuntu SRU process, e.g., each patch has to demonstrably fix a bug, and each patch is vetted by upstream by originating either directly from Linus' tree or in a minimally backported form of that patch.

The 2.6.32.21 upstream stable patch set is now available and contains fixes for this problem. It should be included in the Ubuntu kernel as well.

Related commits are:

a192aa1 (pre-stable) Fix compiling NFS when backporting writeback
d85d9ae (pre-stable) writeback: simplify the write back thread queue
aae03dc (pre-stable) writeback: split writeback_inodes_wb
1c12fa6 (pre-stable) writeback: remove writeback_inodes_wbc
06ff716 (pre-stable) writeback: fix pin_sb_for_writeback
43fa64c (pre-stable) writeback: add missing requeue_io in writeback_inodes_wb
4d214a5 (pre-stable) writeback: simplify and split bdi_start_writeback
4be0607 (pre-stable) writeback: simplify wakeup_flusher_threads
899d625 (pre-stable) writeback: fix writeback_inodes_wb from writeback_inodes_sb
683e9dd (pre-stable) writeback: enforce s_umount locking in writeback_inodes_sb
b3bf7c2 (pre-stable) writeback: queue work on stack in writeback_inodes_sb
5727b1b (pre-stable) writeback: fix writeback completion notifications
fb743c6 (pre-stable) writeback: add missing kernel-doc notation
08e2c5a (pre-stable) vfs: improve writeback_inodes_wb()
47dd2f7 (pre-stable) writeback: remove unused nonblocking and congestion checks
72201eb (pre-stable) writeback: remove the always false bdi_cap_writeback_dirty() test
5e19418 UBUNTU: SAUCE: sync before umount to reduce time taken by ext4 umount

[Martin Pitt (pitti)]

Accepted linux into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[nutznboltz (nutznboltz-deactivatedaccount)]

Also look for "writeback" in

Bug #543617
Bug #585092
Bug #601361
Bug #620242
Bug #628047

[Steve Conklin (sconklin)]

We need some testing of the proposed kernel to see whether this has been resolved - please see comment #35 for information about how to get the proposed kernel.

Thanks

[Chris Siebenmann (cks)]

I can report that the proposed kernel resolves what is either this issue or a closely related
issue. On our Ubuntu 10.04 machines, unmounting an NFS filesystem takes a significant
amount of time, on the order of several seconds to tens of seconds; my test machine runs
between 15 seconds and 25 seconds per filesystem. This happens even with a 'sync' run
immediately before the unmount.

(This happens on both 2.6.32-24-generic-pae i686 and 2.6.32-24-server x86_64. One
unpleasant consequence is that shutting down a machine takes a very long time, as we
have more than 200 NFS mounts and the shutdown scripts do the unmounts serially.)

The proposed kernel solves the problem on at least x86_64; unmounting a NFS
filesystem is almost instant (time says 0.16 seconds real) and the machine shuts
down or reboots immediately when requested.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 2.6.32-25.44

---------------
linux (2.6.32-25.44) lucid-proposed; urgency=low

  [ Brad Figg ]

  * SAUCE: (no-up) Modularize vesafb -- fix initialization
    - LP: #611471

  [ Stefan Bader ]

  * Revert "SAUCE: sync before umount to reduce time taken by ext4 umount"
    - LP: #543617, #585092

  [ Steve Conklin ]

  * Revert "SAUCE: tulip: Let dmfe handle davicom on non-sparc"
    - LP: #607824

  [ Tim Gardner ]

  * [Config] Added ums-cypress to udeb
    - LP: #576066

  [ Upstream Kernel Changes ]

  * Revert "PCI quirk: Disable MSI on VIA K8T890 systems"
    - LP: #607824
  * Revert "PCI quirks: disable msi on AMD rs4xx internal gfx bridges"
    - LP: #607824
  * Revert "(pre-stable) Input: psmouse - reset all types of mice before
    reconnecting"
    - LP: #607824
  * Revert "jbd: jbd-debug and jbd2-debug should be writable"
    - LP: #607824
  * Revert "ext4: Make fsync sync new parent directories in no-journal
    mode"
    - LP: #615548
  * Revert "ext4: Fix compat EXT4_IOC_ADD_GROUP"
    - LP: #615548
  * Revert "ext4: Conditionally define compat ioctl numbers"
    - LP: #615548
  * Revert "ext4: restart ext4_ext_remove_space() after transaction
    restart"
    - LP: #615548
  * Revert "ext4: Clear the EXT4_EOFBLOCKS_FL flag only when warranted"
    - LP: #615548
  * Revert "ext4: Avoid crashing on NULL ptr dereference on a filesystem
    error"
    - LP: #615548
  * Revert "ext4: Use bitops to read/modify i_flags in struct
    ext4_inode_info"
    - LP: #615548
  * Revert "ext4: Show journal_checksum option"
    - LP: #615548
  * Revert "ext4: check for a good block group before loading buddy pages"
    - LP: #615548
  * Revert "ext4: Prevent creation of files larger than RLIMIT_FSIZE using
    fallocate"
    - LP: #615548
  * Revert "ext4: Remove extraneous newlines in ext4_msg() calls"
    - LP: #615548
  * Revert "ext4: init statistics after journal recovery"
    - LP: #615548
  * Revert "ext4: clean up inode bitmaps manipulation in ext4_free_inode"
    - LP: #615548
  * Revert "ext4: Do not zero out uninitialized extents beyond i_size"
    - LP: #615548
  * Revert "ext4: don't scan/accumulate more pages than mballoc will
    allocate"
    - LP: #615548
  * Revert "ext4: stop issuing discards if not supported by device"
    - LP: #615548
  * Revert "ext4: check s_log_groups_per_flex in online resize code"
    - LP: #615548
  * Revert "ext4: fix quota accounting in case of fallocate"
    - LP: #615548
  * Revert "ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode"
    - LP: #615548
  * Revert "ext4: rename ext4_mb_release_desc() to ext4_mb_unload_buddy()"
    - LP: #615548
  * Revert "ext4: Remove unnecessary call to ext4_get_group_desc() in
    mballoc"
    - LP: #615548
  * Revert "ext4: fix memory leaks in error path handling of
    ext4_ext_zeroout()"
    - LP: #615548
  * Revert "ext4: check missed return value in ext4_sync_file()"
    - LP: #615548
  * Revert "ext4: Issue the discard operation *before* releasing the blocks
    to be reused"
    - LP: #615548
  * Revert "ext4: Fix buffer head leaks after calls to
    ext4_get_inode_loc()"
    - LP: #615548
  * Revert "ex...