using ipsec, many connections result in no buffer space error

Bug #1486670 reported by Dan Streetman on 2015-08-19
24
This bug affects 4 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Undecided
Dan Streetman
Precise
Undecided
Dan Streetman
Trusty
Undecided
Dan Streetman
Vivid
Undecided
Dan Streetman
Wily
Undecided
Dan Streetman

Bug Description

Reproduction info:

set up two LXC containers (although this probably isn't specific to LXC containers), and inside each setup ipsec with something similar to:

conn nodeN
aggressive=yes
authby=secret
auto=start
closeaction=restart
dpdaction=restart
esp=aes256-aes256gmac-modp1024
ike=aes256-sha512-modp1024
keyexchange=ikev2
left=10.0.3.145
leftid=10.0.3.145
lifetime=12h
reauth=no
right=10.0.3.199
type=transport

then repeatedly open connections to the peer, e.g.:

while true; do ping -c1 10.0.3.199 ; sleep 0.1 ; done

eventually, the connections will fail with:

connect: No buffer space available

the reproduction can be sped up by reducing the xfrm4_gc_thresh, e.g.:

echo 5 > /proc/sys/net/ipv4/xfrm4_gc_thresh

Once the error occurs, no more connections can be made to the peer (all fail with no buffer space available), however after a long period (e.g. overnight) the buffers will be cleaned up and connections can be made again.

this happens even on the latest net-next kernel.

Dan Streetman (ddstreet) on 2015-08-19
Changed in linux (Ubuntu):
assignee: nobody → Dan Streetman (ddstreet)
status: New → In Progress
Dan Streetman (ddstreet) on 2015-08-28
tags: added: sts
Dan Streetman (ddstreet) wrote :

This is caused by a bug that appears to have been present since ~2008. Proposed upstream patch:
http://marc.info/?l=linux-netdev&m=144596262420164&w=2

Dan Streetman (ddstreet) on 2015-10-30
Changed in linux (Ubuntu Precise):
assignee: nobody → Dan Streetman (ddstreet)
Changed in linux (Ubuntu Trusty):
assignee: nobody → Dan Streetman (ddstreet)
Changed in linux (Ubuntu Wily):
assignee: nobody → Dan Streetman (ddstreet)
Changed in linux (Ubuntu Precise):
status: New → In Progress
Changed in linux (Ubuntu Trusty):
status: New → In Progress
Changed in linux (Ubuntu Wily):
status: New → In Progress
Dan Streetman (ddstreet) wrote :

Short summary:

ipsec uses a struct dst_ops object per net-namespace (e.g. per container), but does not correctly initialize each dst_ops object's percpu counter. This results in incorrect values for each net namespace's dst_ops counter.

Full details:

ipsec uses xfrm objects, which contain dst objects, which are tracked via the xfrm dst_ops struct in its percpu counter. However, ipsec creates a dst_ops object for every net namespace, not just the main net namespace. A dst_ops template is created, and its contents copied to the dst_ops object for each new net namespace. However, ipsec only initializes the percpu counter in the dst_ops object once - for the template. The way percpu counters work is, the percpu counter object has a main counter variable, and a pointer to the percpu counter variables. The percpu variables only go up to a small "batch" size (32 or so), at which point the percpu variable's count is moved to the main counter variable. However since multiple ipsec net namespaces are all using different main counter variables but the same percpu counter variables, the count from each percpu variable can be moved to a different net namespace. The result is, one net namespace (i.e. container) may have its xfrm count decrease to below 0, while another net namespace may have its xfrm count increase forever, and eventually cause complete ipsec failure once the xfrm4_gc_thresh limit is exceeded.

Dan Streetman (ddstreet) wrote :

This is in the ipsec (git://git.kernel.org/pub/scm/linux/kernel/git/klassert/ipsec.git) tree, but not yet in net-next (git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git). Once it hits net-next, I'll request it for the net stable queue (if needed).

Dan Streetman (ddstreet) wrote :

Patch is now in mainline; I'll request it gets added to net stable.

tags: added: kernel-da-key
Brad Figg (brad-figg) on 2016-02-01
Changed in linux (Ubuntu Trusty):
status: In Progress → Fix Committed
Brad Figg (brad-figg) on 2016-02-01
Changed in linux (Ubuntu Vivid):
status: New → Fix Committed
Brad Figg (brad-figg) on 2016-02-01
Changed in linux (Ubuntu Wily):
status: In Progress → Fix Committed
Dan Streetman (ddstreet) on 2016-02-05
Changed in linux (Ubuntu Precise):
status: In Progress → Invalid
Brad Figg (brad-figg) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-trusty' to 'verification-done-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-trusty
tags: added: verification-needed-vivid
Brad Figg (brad-figg) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-vivid' to 'verification-done-vivid'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-wily
Brad Figg (brad-figg) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-wily' to 'verification-done-wily'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

I'm still able to duplicate this bug using:

linux-image-3.13.0-78-generic (from trusty-backports)
linux-image-3.19.0-50-generic (from linux-image-generic-lts-vivid)

The LXC images failed to start under linux-image-4.2.0-28-generic, with a kernel oops.

I also tried, in Xenial, linux-image-4.4.0-2-generic and that failed.

Setting /proc/sys/net/ipv4/xfrm4_gc_thresh to 5 causes the failure almost immediately.

I would like to confirm my procedure however. I've been changing /proc/sys/net/ipv4/xfrm4_gc_thresh inside the containers, not the host. Is this correct?

Dan Streetman (ddstreet) wrote :
Download full text (3.8 KiB)

> The LXC images failed to start under linux-image-4.2.0-28-generic, with a kernel oops.

this bug isn't about kernel oopses.

> Setting /proc/sys/net/ipv4/xfrm4_gc_thresh to 5 causes the failure almost immediately.
>
> I would like to confirm my procedure however. I've been changing /proc/sys/net/ipv4/xfrm4_gc_thresh inside the containers,
> not the host. Is this correct?

no, that's not correct, and unfortunately the "reproducer" in this bug description is completely invalid (it was copied from a private bug for this issue). ipsec will NEVER work with gc_thresh set to 5, due to the internal details between the xfrm gc_thresh and the hardcoded flowcache per-cpu hashtable size limit. In upstream the xfrm4_gc_thresh has been changed to INT_MAX, because using a gc_thresh doesn't make any sense for xfrm dst entries; the total number of them possible is entirely dependent on the number of cpus in the system.

There is 1 xfrm dst entry per flowcache entry, and the flowcache entries are kept in a per-cpu hashtable that is strictly limited at 4096 entries (per cpu). So, the total number of xfrm dst entries will be 4096 * num_active_cpus(), meaning that since the dst code stops allowing new dst allocation once the dst entry count is >= 2 * gc_thresh, the threshold (with the current default 32k gc_thresh) where dst allocation failures can begin to be seen is (32k * 2) / 4096 = 16 cpus. On systems with less than 16 cpus, at the default gc_thresh of 32k, there will never be any dst allocation failures (except due to the real bug this addresses). On systems with 16 or more cpus, with the default gc_thresh of 32k, there will/can be dst alloc failures (with a high enough ipsec usage rate, creating new connections - the flowcache clears all its entries every 10 minutes, so a lightly loaded ipsec with > 16 cpus could be fine).

Setting the xfrm4_gc_thresh value to anything less than (4k * 2 * CPUS) will result in failures, and in fact there is no point in setting xfrm4_gc_thresh to ANYTHING other than INT_MAX because it doesn't actually remove any dst entries.

All that, unfortunately, is actually tangential to this real bug - the problem here is with multiple net namespaces (i.e. multiple containers) all running ipsec, the dst entry counter changes for one container can incorrectly be given to a different container - so one container could have its dst entry count steadily go down (incorrectly) while another container's dst entry count keeps going up (incorrectly). The latter container there would eventually reach its 2 * gc_thresh limit and encounter dst allocation failures, making its ipsec network unusable. Unfortunately, that error looks identical to the error when the dst entry counter is correct, but the number of system cpus is > 16 as described above.

To test this fix, multiple containers must be started (just 2 is fine). On each container, new ipsec connections should be created as fast as possible; e.g. something like:

while true ; do ping -c 1 OTHER_CONTAINER ; done

so each container is pinging each other - but it's important to use -c 1 so that each ping creates a new ipsec dst entry; just normal ping will re-use the existing dst e...

Read more...

Dan Streetman (ddstreet) wrote :

> To speed up reproduction of this bug, lower the xfrm4_gc_thresh to a value ABOVE (2 * 4096 * CPUS), but close to it -
> e.g. something like 10k * CPUS

sorry got the math wrong on the verification - the xfrm4_gc_thresh should be set to above ((4096 * CPUS) / 2), so something like 4K * CPUS, or even (2K * CPUS) + 4k; basically just above the max flowcache limit, plus a bit for dst entries that are released but not yet freed/cleaned yet.

And to re-affirm, for production use the xfrm4_gc_thresh should NEVER be set to anything other than INT_MAX (i.e., a number higher than 4k * 2 * CPUS) - there's absolutely no benefit to setting it any lower than max, and a real chance of causing failures if set too low.

Dan Streetman (ddstreet) wrote :

and a reminder - the /proc/sys/net/ipv4/xfrm4_gc_thresh param is a per-netns value, so it should be changed in each container.

Dan Streetman (ddstreet) on 2016-02-09
Changed in linux (Ubuntu Vivid):
assignee: nobody → Dan Streetman (ddstreet)
Dan Streetman (ddstreet) wrote :

> To test this fix, multiple containers must be started (just 2 is fine).

note - it can be reproduced with just 2, but it happens exponentially faster with a higher number of containers.

Dan Streetman (ddstreet) wrote :

the systemtap script below can be used to monitor the dst count for all net namespaces. When any of the counts goes significantly negative (more than 32 * CPUS negative) it indicates this bug is reproduced - meaning, the count from one net namespace was incorrectly shifted to another net namespace, and once that happens enough times one (or more) net namespaces have a count that goes negative (which is not possible). Other net namespaces have counts that are much higher than they should be. Note this script is just for ipv4, but the bug exists for ipv6 also (and the patch fixes ipv6 also).

#!/usr/bin/stap

global dst_count

probe kernel.function("xfrm_resolve_and_create_bundle") {
  if ($family == 2) {
    dst_count[&$pols[0]->xp_net] = $pols[0]->xp_net->xfrm->xfrm4_dst_ops->pcpuc_entries->count
  }
}

probe timer.sec(1) {
  foreach (c in dst_count) {
    printf("%ld ", dst_count[c])
  }
  print("\n")
}

Dan Streetman (ddstreet) wrote :

correction on the script above, to only show each netns count once (script above duplicates netns counts)

 probe kernel.function("xfrm_resolve_and_create_bundle") {
   if ($family == 2) {
- dst_count[&$pols[0]->xp_net] = $pols[0]->xp_net->xfrm->xfrm4_dst_ops->pcpuc_entries->count
+ dst_count[$pols[0]->xp_net->loopback_dev] = $pols[0]->xp_net->xfrm->xfrm4_dst_ops->pcpuc_entries->count
   }
 }

Dan Streetman (ddstreet) on 2016-02-11
tags: added: verification-done-trusty
removed: verification-needed-trusty
Dan Streetman (ddstreet) on 2016-02-17
tags: added: verification-done-vivid
removed: verification-needed-vivid
Dan Streetman (ddstreet) wrote :

and on wily, this patch is required on top of the patch from comment 14:

 probe kernel.function("xfrm_resolve_and_create_bundle") {
   if ($family == 2) {
- dst_count[$pols[0]->xp_net->loopback_dev] = $pols[0]->xp_net->xfrm->xfrm4_dst_ops->pcpuc_entries->count
+ dst_count[$pols[0]->xp_net->net->loopback_dev] = $pols[0]->xp_net->net->xfrm->xfrm4_dst_ops->pcpuc_entries->count
   }
 }

Launchpad Janitor (janitor) wrote :
Download full text (43.7 KiB)

This bug was fixed in the package linux - 4.2.0-30.35

---------------
linux (4.2.0-30.35) wily; urgency=low

  [ Seth Forshee ]

  * SAUCE: cred: Add clone_cred() interface
    - LP: #1531747, #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Use mounter's credentials instead of selectively
    raising caps
    - LP: #1531747, #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Skip permission checking for trusted.overlayfs.*
    xattrs
    - LP: #1531747, #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Be more careful about copying up sxid files
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Propogate nosuid from lower and upper mounts
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576

linux (4.2.0-29.34) wily; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1543167

  [ Brad Figg ]

  * Revert "SAUCE: apparmor: fix sleep from invalid context"
    - LP: #1542049

  [ Upstream Kernel Changes ]

  * Revert "af_unix: Revert 'lock_interruptible' in stream receive code"
    - LP: #1540731

linux (4.2.0-28.33) wily; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
    - LP: #1540634

  [ Brad Figg ]

  * CONFIG: CONFIG_DEBUG_UART_BCM63XX is not set

  [ J. R. Okajima ]

  * SAUCE: ubuntu: aufs: tiny, extract a new func xino_fwrite_wkq()
    - LP: #1533043
  * SAUCE: ubuntu: aufs: for 4.3, XINO handles EINTR from the dying process
    - LP: #1533043

  [ John Johansen ]

  * SAUCE: (no-up): apparmor: fix for failed mediation of socket that is
    being shutdown
    - LP: #1446906
  * SAUCE: apparmor: fix sleep from invalid context
    - LP: #1539349

  [ Tim Gardner ]

  * [Config] Add pvpanic to virtual flavour
    - LP: #1537923

  [ Upstream Kernel Changes ]

  * Revert "ACPI / LPSS: allow to use specific PM domain during ->probe()"
    - LP: #1540532
  * tools: Add a "make all" rule
    - LP: #1536370
  * vf610_adc: Fix internal temperature calculation
    - LP: #1536370
  * iio: lpc32xx_adc: fix warnings caused by enabling unprepared clock
    - LP: #1536370
  * iio:ad5064: Make sure ad5064_i2c_write() returns 0 on success
    - LP: #1536370
  * iio: ad5064: Fix ad5629/ad5669 shift
    - LP: #1536370
  * iio:ad7793: Fix ad7785 product ID
    - LP: #1536370
  * iio: adc: vf610_adc: Fix division by zero error
    - LP: #1536370
  * mmc: mmc: Improve reliability of mmc_select_hs200()
    - LP: #1536370
  * mmc: mmc: Fix HS setting in mmc_select_hs400()
    - LP: #1536370
  * mmc: mmc: Move mmc_switch_status()
    - LP: #1536370
  * mmc: mmc: Improve reliability of mmc_select_hs400()
    - LP: #1536370
  * crypto: qat - don't use userspace pointer
    - LP: #1536370
  * iio: si7020: Swap data byte order
    - LP: #1536370
  * iio: adc: xilinx: Fix VREFN scale
    - LP: #1536370
  * ipmi: Start the timer and thread on internal msgs
    - LP: #1536370
  * drm/i915: quirk backlight present on Macbook 4, 1
    - LP: #1536370
  * drm/i915: get runtime PM reference around GEM set_caching IOCTL
    - LP: #1536370
  * drm/radeon: Disable uncacheable CPU mappings of GTT with RV6xx
    - LP: #1536370
  *...

Changed in linux (Ubuntu Wily):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :
Download full text (12.3 KiB)

This bug was fixed in the package linux - 3.19.0-51.57

---------------
linux (3.19.0-51.57) vivid; urgency=low

  [ Seth Forshee ]

  * SAUCE: cred: Add clone_cred() interface
    - LP: #1531747, #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Use mounter's credentials instead of selectively
    raising caps
    - LP: #1531747, #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Skip permission checking for trusted.overlayfs.*
    xattrs
    - LP: #1531747, #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Be more careful about copying up sxid files
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Propogate nosuid from lower and upper mounts
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576

linux (3.19.0-50.56) vivid; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
    - LP: #1540576

  [ J. R. Okajima ]

  * SAUCE: ubuntu: aufs: tiny, extract a new func xino_fwrite_wkq()
    - LP: #1533043
  * SAUCE: ubuntu: aufs: for 4.3, XINO handles EINTR from the dying process
    - LP: #1533043

  [ John Johansen ]

  * SAUCE: (no-up): apparmor: fix for failed mediation of socket that is
    being shutdown
    - LP: #1446906

  [ Upstream Kernel Changes ]

  * drivers/base/memory.c: fix kernel warning during memory hotplug on
    ppc64
    - LP: #1463654
  * sched/wait: Fix signal handling in bit wait helpers
    - LP: #1537859
  * sched/wait: Fix the signal handling fix
    - LP: #1537859
  * ARC: Fix silly typo in MAINTAINERS file
    - LP: #1537859
  * ip6mr: call del_timer_sync() in ip6mr_free_table()
    - LP: #1537859
  * gre6: allow to update all parameters via rtnl
    - LP: #1537859
  * atl1c: Improve driver not to do order 4 GFP_ATOMIC allocation
    - LP: #1537859
  * sctp: use the same clock as if sock source timestamps were on
    - LP: #1537859
  * sctp: update the netstamp_needed counter when copying sockets
    - LP: #1537859
  * sctp: also copy sk_tsflags when copying the socket
    - LP: #1537859
  * net: qca_spi: fix transmit queue timeout handling
    - LP: #1537859
  * ipv6: sctp: clone options to avoid use after free
    - LP: #1537859
  * net: add validation for the socket syscall protocol argument
    - LP: #1537859
  * sh_eth: fix kernel oops in skb_put()
    - LP: #1537859
  * net: fix IP early demux races
    - LP: #1537859
  * vlan: Fix untag operations of stacked vlans with REORDER_HEADER off
    - LP: #1537859
  * skbuff: Fix offset error in skb_reorder_vlan_header
    - LP: #1537859
  * pptp: verify sockaddr_len in pptp_bind() and pptp_connect()
    - LP: #1537859
  * bluetooth: Validate socket address length in sco_sock_bind().
    - LP: #1537859
  * fou: clean up socket with kfree_rcu
    - LP: #1537859
  * af_unix: Revert 'lock_interruptible' in stream receive code
    - LP: #1537859
  * KEYS: Fix race between read and revoke
    - LP: #1537859
  * tools: Add a "make all" rule
    - LP: #1537859
  * efi: Disable interrupts around EFI calls, not in the epilog/prolog
    calls
    - LP: #1537859
  * fuse: break infinite loop in fuse_fill_write_pages()
    - LP: #1537859
  * usb: gadget: pxa2...

Changed in linux (Ubuntu Vivid):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :
Download full text (9.5 KiB)

This bug was fixed in the package linux - 3.13.0-79.123

---------------
linux (3.13.0-79.123) trusty; urgency=low

  [ Seth Forshee ]

  * SAUCE: cred: Add clone_cred() interface
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Use mounter's credentials instead of full kernel
    credentials
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Skip permission checking for trusted.overlayfs.*
    xattrs
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Be more careful about copying up sxid files
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Propogate nosuid from lower and upper mounts
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576

linux (3.13.0-78.122) trusty; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
    - LP: #1540559

  [ Eric Dumazet ]

  * SAUCE: (no-up) udp: properly support MSG_PEEK with truncated buffers
    - LP: #1527902

  [ J. R. Okajima ]

  * SAUCE: ubuntu: aufs: tiny, extract a new func xino_fwrite_wkq()
    - LP: #1533043
  * SAUCE: ubuntu: aufs: for 4.3, XINO handles EINTR from the dying process
    - LP: #1533043

  [ Upstream Kernel Changes ]

  * Revert "[stable-only] net: add length argument to
    skb_copy_and_csum_datagram_iovec"
    - LP: #1538756
  * unregister_netdevice : move RTM_DELLINK to until after ndo_uninit
    - LP: #1525324
  * rtnetlink: delay RTM_DELLINK notification until after ndo_uninit()
    - LP: #1525324
  * Drivers: hv: Eliminate the channel spinlock in the callback path
    - LP: #1519897
  * Drivers: hv: vmbus: Implement per-CPU mapping of relid to channel
    - LP: #1519897
  * Drivers: hv: vmbus: Suport an API to send pagebuffers with additional
    control
    - LP: #1519897
  * Drivers: hv: vmbus: Suport an API to send packet with additional
    control
    - LP: #1519897
  * Drivers: hv: vmbus: Export the vmbus_sendpacket_pagebuffer_ctl()
    - LP: #1519897
  * Drivers: hv: vmbus: Fix a siganlling host signalling issue
    - LP: #1519897
  * Drivers: hv: vmbus: Fix a Host signaling bug
    - LP: #1519897
  * ARC: Fix silly typo in MAINTAINERS file
    - LP: #1538756
  * ip6mr: call del_timer_sync() in ip6mr_free_table()
    - LP: #1538756
  * gre6: allow to update all parameters via rtnl
    - LP: #1538756
  * atl1c: Improve driver not to do order 4 GFP_ATOMIC allocation
    - LP: #1538756
  * sctp: use the same clock as if sock source timestamps were on
    - LP: #1538756
  * sctp: update the netstamp_needed counter when copying sockets
    - LP: #1538756
  * ipv6: sctp: clone options to avoid use after free
    - LP: #1538756
  * net: add validation for the socket syscall protocol argument
    - LP: #1538756
  * sh_eth: fix kernel oops in skb_put()
    - LP: #1538756
  * pptp: verify sockaddr_len in pptp_bind() and pptp_connect()
    - LP: #1538756
  * bluetooth: Validate socket address length in sco_sock_bind().
    - LP: #1538756
  * af_unix: Revert 'lock_interruptible' in stream receive code
    - LP: #1538756
  * KEYS: Fix race between read and revoke
    - LP: #1538756
  * tools: Add a "make all" rule
    - LP: #1538...

Read more...

Changed in linux (Ubuntu Trusty):
status: Fix Committed → Fix Released
status: Fix Committed → Fix Released
Dan Streetman (ddstreet) on 2016-02-24
tags: added: verification-done-wily
removed: verification-needed-wily
Dan Streetman (ddstreet) on 2016-02-24
Changed in linux (Ubuntu):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers