AUFS can hang up; Please update to v20160111

[Akihiro Suda]

AUFS can hang up (more precisely, produces unkillable processes) with some kernels that contain the commit 296291cdd1629c308114504b850dc343eabc278.
- 14.04LTS: http://kernel.ubuntu.com/git/ubuntu/ubuntu-trusty.git/commit/?id=b8702f3cf9c06914476fb3206039131c2c3d8450
- 15.04: http://kernel.ubuntu.com/git/ubuntu/ubuntu-vivid.git/commit/?id=6b08592b8acc677d5b9bb7986343fdd6e0ad3303
- 15.10: http://kernel.ubuntu.com/git/ubuntu/ubuntu-wily.git/commit/?id=4d291df30fb7a94d13c6d38addf8d85d38f0111b

Many docker users have been experiencing this issue: https://github.com/docker/docker/issues/18180

The issue has been fixed since AUFS v20160111: http://permalink.gmane.org/gmane.linux.file-systems.aufs.user/5345
So I suggest updating AUFS to v20160111.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1533043

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Akihiro Suda (suda-kyoto)]

As the bug has been diagnosed well, I think apport-collect is not needed.
Please let me know if needed.

[Mike Atlas (mikeatlas)]

For what it's worth, Dockerized containers running common open-source applications (Cassandra, Mongo, Flynn) have stumbled hard on this - and of course, it's not just on Ubuntu, but all Linux kernels with 296291cdd: https://bugzilla.kernel.org/show_bug.cgi?id=109971 -

Akihiro has been a champion in tracking down the cause, reproduction of the problem, a patch, and pushing for a fix in all the right places. Give this guy a high five or a beer if you know him in person, he's helping a lot of people out with his efforts.

[Yucong Sun (sunyucong)]

Hello,

What is the ETA for this fix to be released?

Thanks!

[ConMotto (conmotto)]

Anyone? A *lot* of people are being affected by this, unfortunately.

[Dave Chiluk (chiluk)]

The upstream fix for this issue is available here.

https://github.com/sfjro/aufs4-linux/commit/5e439ff30c92143d9a9ee3401a84e34c9852533b

I will work on building test kernels tomorrow once I've had a few hours of sleep.

[Dave Chiluk (chiluk)]

Thanks apw for the backport and pushing the patches to the kernel-team mailing list.

I have packaged up apw's backport for the trusty kernels, and put them in a ppa for testing
https://launchpad.net/~chiluk/+archive/ubuntu/1533043/+packages
I will push kernels for vivid and wily to that ppa as well shortly.

I would greatly appreciate if I could get some testing from the many users on this bug verifying that the kernels in the ppa resolve the issue. When responding please indicate which kernel you tested *(3.13, 3.16, 3.19, 4.2). Additionally if someone has already tested the kernel you are testing there is no need to report that here unless your testing conflicts with the earlier report *(i.e. someone says it works, but it fails for you).

Also please remember to install both the linux-image package and the linux-image-extra package as both are required for baremetal installations. If you have dkms modules please remember to install the -headers package as well.

That being said the kernels are still building as of this writing, and should probably take another hour or two as they are being built with debug packages enabled just in case..

Thank you,

[Diane Brent (drbrent)]

Looks like this is in the current sru cycle per my chat with Hohnbaum. Should be released Feb 20.
Thank you!

[Thomas Recloux (thomas-recloux)]

I tested the ppa Kernels :

3.13.0-77.121hf1533043v20160201b1 => OK
3.16.0-60.80~14.04.1hf1533043v20160201b1 => OK
3.19.0-49.55~14.04.1hf1533043v20160201b1 => OK
4.2.0-27.32~14.04.1hf1533043v20160201b1 => OK

Thanks .

[Thomas Recloux (thomas-recloux)]

3.19.0-49.55hf1533043v20160201b1 => OK

4.2.0-27.32hf1533043v20160201b1 => OK

[Dave Chiluk (chiluk)]

Fantastic! Thank you Thomas. That was really helpful.

[Ryan Sundberg (ryan-sundberg)]

Tested 3.19.0-49-generic #55~14.04.1hf1533043v20160201b1-Ubuntu on bare metal.

Works like a charm. What a great effort by everyone involved. Thanks a lot!

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-trusty' to 'verification-done-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-vivid' to 'verification-done-vivid'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-wily' to 'verification-done-wily'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Richard Hansen (rhansen)]

Would someone please provide a simple test case to verify that the version in -proposed fixes the bug?

[Richard Hansen (rhansen)]

I'm unable to reproduce the problem on wily (I'm actually running wily, not trusty with the wily kernel), so I changed the verification-needed-wily tag to verification-done-wily.

[Maxim N. Kostrikin (mkostrikin)]

Running proposed wily kernel 4.2.0-28-generic on trusty fix the issue,

Status: Downloaded newer image for akihirosuda/test18180:latest
[INFO] Checking whether hitting docker#18180.
....................................................................................................
[INFO] OK. not hitting docker#18180.
[INFO] Checking whether sendfile(2) is killable.
[INFO] If the container hangs up here, you are still facing the bug that linux@296291cd tried to fix.
/test.sh: line 22: 1411 Killed /sendfile-test

I have not idea, where to put tag and what kind of tag

[Bryan Quigley (bryanquigley)]

@mkostrikin
That provides another confirmation like from comment #17 about the 4.2.0 kernel/wily.

We could still use testing for:
3.13.0-78 (on trusty - Non-HWE)
3.16.0-61 (on trusty - utopic-HWE - linux-image-generic-lts-utopic)
3.19.0-50 (on trusty - vivid-HWE - linux-image-generic-lts-vivid)

[David Röthlisberger (drothlis)]

@rhansen: according to [1] the testcase is:

    docker run -it --rm akihirosuda/test18180

(The above will hang forever with the bad kernel.)

@brad-figg: I confirm that 3.13.0-77 fails the above test, and that
3.13.0-78 passes.

I also have a private testcase that was triggering this bug 1/8 times
with 3.13.0-77; it passed 70 times with 3.13.0-78 with no failures.

[1]: https://github.com/docker/docker/issues/18180#issuecomment-167192392

[Richard Hansen (rhansen)]

Running:

    docker run -it --rm akihirosuda/test18180

I was able to verify the last two unverified kernels:

3.16.0-61 (utopic kernel on trusty via linux-image-generic-lts-utopic): verified
3.19.0-50 (vivid kernel on trusty via linux-image-generic-lts-vivid): verified

So it looks like everything is good to go.

[Chris Lasher (chris.lasher)]

Is this patch also going to be included in an update for the linux-generic-lts-trusty package for Ubuntu 12.04 LTS?

[Dave Chiluk (chiluk)]

Yes, it is available in the following kernel version
3.13.0-78
3.16.0-61
3.19.0-50
4.2.0-28

I am now pulling my PPA as the kernels in -proposed contain this fix.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.2.0-30.35

---------------
linux (4.2.0-30.35) wily; urgency=low

  [ Seth Forshee ]

  * SAUCE: cred: Add clone_cred() interface
    - LP: #1531747, #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Use mounter's credentials instead of selectively
    raising caps
    - LP: #1531747, #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Skip permission checking for trusted.overlayfs.*
    xattrs
    - LP: #1531747, #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Be more careful about copying up sxid files
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Propogate nosuid from lower and upper mounts
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576

linux (4.2.0-29.34) wily; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1543167

  [ Brad Figg ]

  * Revert "SAUCE: apparmor: fix sleep from invalid context"
    - LP: #1542049

  [ Upstream Kernel Changes ]

  * Revert "af_unix: Revert 'lock_interruptible' in stream receive code"
    - LP: #1540731

linux (4.2.0-28.33) wily; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
    - LP: #1540634

  [ Brad Figg ]

  * CONFIG: CONFIG_DEBUG_UART_BCM63XX is not set

  [ J. R. Okajima ]

  * SAUCE: ubuntu: aufs: tiny, extract a new func xino_fwrite_wkq()
    - LP: #1533043
  * SAUCE: ubuntu: aufs: for 4.3, XINO handles EINTR from the dying process
    - LP: #1533043

  [ John Johansen ]

  * SAUCE: (no-up): apparmor: fix for failed mediation of socket that is
    being shutdown
    - LP: #1446906
  * SAUCE: apparmor: fix sleep from invalid context
    - LP: #1539349

  [ Tim Gardner ]

  * [Config] Add pvpanic to virtual flavour
    - LP: #1537923

  [ Upstream Kernel Changes ]

  * Revert "ACPI / LPSS: allow to use specific PM domain during ->probe()"
    - LP: #1540532
  * tools: Add a "make all" rule
    - LP: #1536370
  * vf610_adc: Fix internal temperature calculation
    - LP: #1536370
  * iio: lpc32xx_adc: fix warnings caused by enabling unprepared clock
    - LP: #1536370
  * iio:ad5064: Make sure ad5064_i2c_write() returns 0 on success
    - LP: #1536370
  * iio: ad5064: Fix ad5629/ad5669 shift
    - LP: #1536370
  * iio:ad7793: Fix ad7785 product ID
    - LP: #1536370
  * iio: adc: vf610_adc: Fix division by zero error
    - LP: #1536370
  * mmc: mmc: Improve reliability of mmc_select_hs200()
    - LP: #1536370
  * mmc: mmc: Fix HS setting in mmc_select_hs400()
    - LP: #1536370
  * mmc: mmc: Move mmc_switch_status()
    - LP: #1536370
  * mmc: mmc: Improve reliability of mmc_select_hs400()
    - LP: #1536370
  * crypto: qat - don't use userspace pointer
    - LP: #1536370
  * iio: si7020: Swap data byte order
    - LP: #1536370
  * iio: adc: xilinx: Fix VREFN scale
    - LP: #1536370
  * ipmi: Start the timer and thread on internal msgs
    - LP: #1536370
  * drm/i915: quirk backlight present on Macbook 4, 1
    - LP: #1536370
  * drm/i915: get runtime PM reference around GEM set_caching IOCTL
    - LP: #1536370
  * drm/radeon: Disable uncacheable CPU mappings of GTT with RV6xx
    - LP: #1536370
  *...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.19.0-51.57

---------------
linux (3.19.0-51.57) vivid; urgency=low

  [ Seth Forshee ]

  * SAUCE: cred: Add clone_cred() interface
    - LP: #1531747, #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Use mounter's credentials instead of selectively
    raising caps
    - LP: #1531747, #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Skip permission checking for trusted.overlayfs.*
    xattrs
    - LP: #1531747, #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Be more careful about copying up sxid files
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Propogate nosuid from lower and upper mounts
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576

linux (3.19.0-50.56) vivid; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
    - LP: #1540576

  [ J. R. Okajima ]

  * SAUCE: ubuntu: aufs: tiny, extract a new func xino_fwrite_wkq()
    - LP: #1533043
  * SAUCE: ubuntu: aufs: for 4.3, XINO handles EINTR from the dying process
    - LP: #1533043

  [ John Johansen ]

  * SAUCE: (no-up): apparmor: fix for failed mediation of socket that is
    being shutdown
    - LP: #1446906

  [ Upstream Kernel Changes ]

  * drivers/base/memory.c: fix kernel warning during memory hotplug on
    ppc64
    - LP: #1463654
  * sched/wait: Fix signal handling in bit wait helpers
    - LP: #1537859
  * sched/wait: Fix the signal handling fix
    - LP: #1537859
  * ARC: Fix silly typo in MAINTAINERS file
    - LP: #1537859
  * ip6mr: call del_timer_sync() in ip6mr_free_table()
    - LP: #1537859
  * gre6: allow to update all parameters via rtnl
    - LP: #1537859
  * atl1c: Improve driver not to do order 4 GFP_ATOMIC allocation
    - LP: #1537859
  * sctp: use the same clock as if sock source timestamps were on
    - LP: #1537859
  * sctp: update the netstamp_needed counter when copying sockets
    - LP: #1537859
  * sctp: also copy sk_tsflags when copying the socket
    - LP: #1537859
  * net: qca_spi: fix transmit queue timeout handling
    - LP: #1537859
  * ipv6: sctp: clone options to avoid use after free
    - LP: #1537859
  * net: add validation for the socket syscall protocol argument
    - LP: #1537859
  * sh_eth: fix kernel oops in skb_put()
    - LP: #1537859
  * net: fix IP early demux races
    - LP: #1537859
  * vlan: Fix untag operations of stacked vlans with REORDER_HEADER off
    - LP: #1537859
  * skbuff: Fix offset error in skb_reorder_vlan_header
    - LP: #1537859
  * pptp: verify sockaddr_len in pptp_bind() and pptp_connect()
    - LP: #1537859
  * bluetooth: Validate socket address length in sco_sock_bind().
    - LP: #1537859
  * fou: clean up socket with kfree_rcu
    - LP: #1537859
  * af_unix: Revert 'lock_interruptible' in stream receive code
    - LP: #1537859
  * KEYS: Fix race between read and revoke
    - LP: #1537859
  * tools: Add a "make all" rule
    - LP: #1537859
  * efi: Disable interrupts around EFI calls, not in the epilog/prolog
    calls
    - LP: #1537859
  * fuse: break infinite loop in fuse_fill_write_pages()
    - LP: #1537859
  * usb: gadget: pxa2...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.13.0-79.123

---------------
linux (3.13.0-79.123) trusty; urgency=low

  [ Seth Forshee ]

  * SAUCE: cred: Add clone_cred() interface
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Use mounter's credentials instead of full kernel
    credentials
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Skip permission checking for trusted.overlayfs.*
    xattrs
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Be more careful about copying up sxid files
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Propogate nosuid from lower and upper mounts
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576

linux (3.13.0-78.122) trusty; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
    - LP: #1540559

  [ Eric Dumazet ]

  * SAUCE: (no-up) udp: properly support MSG_PEEK with truncated buffers
    - LP: #1527902

  [ J. R. Okajima ]

  * SAUCE: ubuntu: aufs: tiny, extract a new func xino_fwrite_wkq()
    - LP: #1533043
  * SAUCE: ubuntu: aufs: for 4.3, XINO handles EINTR from the dying process
    - LP: #1533043

  [ Upstream Kernel Changes ]

  * Revert "[stable-only] net: add length argument to
    skb_copy_and_csum_datagram_iovec"
    - LP: #1538756
  * unregister_netdevice : move RTM_DELLINK to until after ndo_uninit
    - LP: #1525324
  * rtnetlink: delay RTM_DELLINK notification until after ndo_uninit()
    - LP: #1525324
  * Drivers: hv: Eliminate the channel spinlock in the callback path
    - LP: #1519897
  * Drivers: hv: vmbus: Implement per-CPU mapping of relid to channel
    - LP: #1519897
  * Drivers: hv: vmbus: Suport an API to send pagebuffers with additional
    control
    - LP: #1519897
  * Drivers: hv: vmbus: Suport an API to send packet with additional
    control
    - LP: #1519897
  * Drivers: hv: vmbus: Export the vmbus_sendpacket_pagebuffer_ctl()
    - LP: #1519897
  * Drivers: hv: vmbus: Fix a siganlling host signalling issue
    - LP: #1519897
  * Drivers: hv: vmbus: Fix a Host signaling bug
    - LP: #1519897
  * ARC: Fix silly typo in MAINTAINERS file
    - LP: #1538756
  * ip6mr: call del_timer_sync() in ip6mr_free_table()
    - LP: #1538756
  * gre6: allow to update all parameters via rtnl
    - LP: #1538756
  * atl1c: Improve driver not to do order 4 GFP_ATOMIC allocation
    - LP: #1538756
  * sctp: use the same clock as if sock source timestamps were on
    - LP: #1538756
  * sctp: update the netstamp_needed counter when copying sockets
    - LP: #1538756
  * ipv6: sctp: clone options to avoid use after free
    - LP: #1538756
  * net: add validation for the socket syscall protocol argument
    - LP: #1538756
  * sh_eth: fix kernel oops in skb_put()
    - LP: #1538756
  * pptp: verify sockaddr_len in pptp_bind() and pptp_connect()
    - LP: #1538756
  * bluetooth: Validate socket address length in sco_sock_bind().
    - LP: #1538756
  * af_unix: Revert 'lock_interruptible' in stream receive code
    - LP: #1538756
  * KEYS: Fix race between read and revoke
    - LP: #1538756
  * tools: Add a "make all" rule
    - LP: #1538...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-lts-utopic - 3.16.0-62.82~14.04.1

---------------
linux-lts-utopic (3.16.0-62.82~14.04.1) trusty; urgency=low

  [ Seth Forshee ]

  * SAUCE: cred: Add clone_cred() interface
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Use mounter's credentials instead of selectively
    raising caps
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Skip permission checking for trusted.overlayfs.*
    xattrs
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Be more careful about copying up sxid files
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Propogate nosuid from lower and upper mounts
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576

linux-lts-utopic (3.16.0-61.81~14.04.1) trusty; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
    - LP: #1541076

  [ Eric Dumazet ]

  * SAUCE: (no-up) udp: properly support MSG_PEEK with truncated buffers
    - LP: #1527902

  [ J. R. Okajima ]

  * SAUCE: ubuntu: aufs: tiny, extract a new func xino_fwrite_wkq()
    - LP: #1533043
  * SAUCE: ubuntu: aufs: for 4.3, XINO handles EINTR from the dying process
    - LP: #1533043

  [ Upstream Kernel Changes ]

  * Revert "[stable-only] net: add length argument to
    skb_copy_and_csum_datagram_iovec"
    - LP: #1540412
  * xfrm: dst_entries_init() per-net dst_ops
    - LP: #1486670
  * fuse: break infinite loop in fuse_fill_write_pages()
    - LP: #1540412
  * usb: gadget: pxa27x: fix suspend callback
    - LP: #1540412
  * iio: fix some warning messages
    - LP: #1540412
  * USB: cp210x: Remove CP2110 ID from compatibility list
    - LP: #1540412
  * USB: cdc_acm: Ignore Infineon Flash Loader utility
    - LP: #1540412
  * USB: serial: Another Infineon flash loader USB ID
    - LP: #1540412
  * ext4: Fix handling of extended tv_sec
    - LP: #1540412
  * jbd2: Fix unreclaimed pages after truncate in data=journal mode
    - LP: #1540412
  * drm/ttm: Fixed a read/write lock imbalance
    - LP: #1540412
  * AHCI: Fix softreset failed issue of Port Multiplier
    - LP: #1540412
  * sata_sil: disable trim
    - LP: #1540412
  * usb-storage: Fix scsi-sd failure "Invalid field in cdb" for USB adapter
    JMicron
    - LP: #1540412
  * staging: lustre: echo_copy.._lsm() dereferences userland pointers
    directly
    - LP: #1540412
  * irqchip/versatile-fpga: Fix PCI IRQ mapping on Versatile PB
    - LP: #1540412
  * usb: core : hub: Fix BOS 'NULL pointer' kernel panic
    - LP: #1540412
  * USB: whci-hcd: add check for dma mapping error
    - LP: #1540412
  * usb: Use the USB_SS_MULT() macro to decode burst multiplier for log
    message
    - LP: #1540412
  * dm btree: fix leak of bufio-backed block in btree_split_sibling error
    path
    - LP: #1540412
  * perf: Fix PERF_EVENT_IOC_PERIOD deadlock
    - LP: #1540412
  * usb: xhci: fix config fail of FS hub behind a HS hub with MTT
    - LP: #1540412
  * ALSA: rme96: Fix unexpected volume reset after rate changes
    - LP: #1540412
  * ALSA: hda - Add inverted dmic for Packard Bell DOTS
    - LP: #1523232, #1540412
  * v...

[Akihiro Suda (suda-kyoto)]

Thank you for releasing the new kernel packages!

[dann frazier (dannf)]

I can see the patch from comment #6 in the xenial kernels starting with Ubuntu-4.4.0-0.12, so going ahead and marking this Fix Released.