Activity log for bug #1640786

Date Who What changed Old value New value Message
2016-11-10 13:27:24 Eric Desrochers bug added bug
2016-11-10 13:30:09 Brad Figg linux (Ubuntu): status New Incomplete
2016-11-10 13:52:12 Eric Desrochers tags sts
2016-11-10 13:53:43 Eric Desrochers description Explanation : It has been brought to my attention that Ubuntu kernel 4.4 has a severe netfilter regression affecting the performance of "/sbin/iptables" command, especially when adding large number of policies. My source have documented everything here[2]. I was able to reproduce the situation on my side, and a kernel bisect identified the same offending commit[1] as my source found for this bug. Running the commit right before the offending one have proven to have expected performance : # commit [71ae0dff] <== Offending commit real 0m33.314s user 0m1.520s sys 0m26.192s # commit [d7b59742] <== Right before offending commit real 0m5.952s user 0m0.124s sys 0m0.220s Reproducer : $ iptables -F $ echo 3 > /proc/sys/vm/drop_caches $ time (./list-addrs 3000 | xargs -n1 iptables -A FORWARD -j ACCEPT -s) "list-addrs" script can be found here[3] Note : * "iptables-restore" doesn't suffer of that netfilter regression, and I'm also aware that "iptables-restore" is the favourite approach since it is way more efficient that iptables that is executed over and over, once for each policy one want to set, but since "/sbin/iptables" takes vastly longer to perform with that commit, I think this need to be address anyway. * I also tried with the latest and greatest iptables upstream code, and got the same result. Reference : [1] - https://github.com/torvalds/linux/commit/71ae0dff02d756e4d2ca710b79f2ff5390029a5f [2] - https://gist.github.com/williammartin/b75e3faf5964648299e4d985413e6c0c [3] - https://gist.github.com/williammartin/b75e3faf5964648299e4d985413e6c0c#file-list-addrs Explanation : It has been brought to my attention that Ubuntu kernel 4.4 has a severe netfilter regression affecting the performance of "/sbin/iptables" command, especially when adding large number of policies. My source have documented everything here[2]. I was able to reproduce the situation on my side, and a kernel bisect identified the same offending commit[1] as my source found for this bug. Running the commit right before the offending one have proven to have expected performance : # commit [71ae0dff] <== Offending commit real 0m33.314s user 0m1.520s sys 0m26.192s # commit [d7b59742] <== Right before offending commit real 0m5.952s user 0m0.124s sys 0m0.220s Reproducer : $ iptables -F $ echo 3 > /proc/sys/vm/drop_caches $ time (./list-addrs 3000 | xargs -n1 iptables -A FORWARD -j ACCEPT -s) "list-addrs" script can be found here[3] Note : * "iptables-restore" doesn't suffer of that netfilter regression, and I'm also aware that "iptables-restore" is the favourite approach since it is way more efficient than iptables that is executed over and over, once for each policy one want to set, but since "/sbin/iptables" takes vastly longer to perform with that commit, I think this need to be address anyway. * I also tried with the latest and greatest iptables upstream code, and got the same result. Reference : [1] - https://github.com/torvalds/linux/commit/71ae0dff02d756e4d2ca710b79f2ff5390029a5f [2] - https://gist.github.com/williammartin/b75e3faf5964648299e4d985413e6c0c [3] - https://gist.github.com/williammartin/b75e3faf5964648299e4d985413e6c0c#file-list-addrs
2016-11-10 14:00:06 Eric Desrochers linux (Ubuntu): status Incomplete Confirmed
2016-11-10 14:00:17 Eric Desrochers linux (Ubuntu): importance Undecided Medium
2016-11-10 20:59:04 Eric Desrochers summary netfilter regression introducing a performance slowdown in iptables netfilter regression introducing a performance slowdown in binary ip/ip6tables
2016-11-11 00:08:39 Jay Vosburgh bug added subscriber Jay Vosburgh
2016-11-14 20:05:06 Joseph Salisbury tags sts kernel-da-key sts
2016-11-16 14:24:13 Eric Desrochers linux (Ubuntu): assignee Eric Desrochers (slashd)
2016-11-21 23:03:35 Eric Desrochers summary netfilter regression introducing a performance slowdown in binary ip/ip6tables netfilter regression introducing a performance slowdown in binary arp/iptables/ip6tables
2016-11-21 23:03:49 Eric Desrochers summary netfilter regression introducing a performance slowdown in binary arp/iptables/ip6tables netfilter regression introducing a performance slowdown in binary arp/ip/ip6tables
2016-11-22 01:27:59 Eric Desrochers linux (Ubuntu): status Confirmed In Progress
2016-11-28 20:53:05 Joseph Salisbury nominated for series Ubuntu Trusty
2016-11-28 20:53:05 Joseph Salisbury bug task added linux (Ubuntu Trusty)
2016-11-28 20:53:13 Joseph Salisbury linux (Ubuntu Trusty): status New In Progress
2016-11-28 20:53:15 Joseph Salisbury linux (Ubuntu Trusty): importance Undecided Medium
2016-11-28 20:54:35 Eric Desrochers linux (Ubuntu Trusty): assignee Eric Desrochers (slashd)
2016-11-28 20:57:27 Joseph Salisbury bug task deleted linux (Ubuntu Trusty)
2016-11-28 20:57:34 Joseph Salisbury nominated for series Ubuntu Xenial
2016-11-28 20:57:34 Joseph Salisbury bug task added linux (Ubuntu Xenial)
2016-11-28 20:57:41 Joseph Salisbury linux (Ubuntu Xenial): importance Undecided Medium
2016-11-28 20:57:44 Joseph Salisbury linux (Ubuntu Xenial): status New In Progress
2016-11-28 20:58:06 Eric Desrochers linux (Ubuntu Xenial): assignee Eric Desrochers (slashd)
2016-12-02 15:25:15 Eric Desrochers linux (Ubuntu): importance Medium High
2016-12-02 15:25:18 Eric Desrochers linux (Ubuntu Xenial): importance Medium High
2016-12-06 12:56:40 Mike Dalessio bug added subscriber Mike Dalessio
2016-12-06 20:06:55 Eric Desrochers description Explanation : It has been brought to my attention that Ubuntu kernel 4.4 has a severe netfilter regression affecting the performance of "/sbin/iptables" command, especially when adding large number of policies. My source have documented everything here[2]. I was able to reproduce the situation on my side, and a kernel bisect identified the same offending commit[1] as my source found for this bug. Running the commit right before the offending one have proven to have expected performance : # commit [71ae0dff] <== Offending commit real 0m33.314s user 0m1.520s sys 0m26.192s # commit [d7b59742] <== Right before offending commit real 0m5.952s user 0m0.124s sys 0m0.220s Reproducer : $ iptables -F $ echo 3 > /proc/sys/vm/drop_caches $ time (./list-addrs 3000 | xargs -n1 iptables -A FORWARD -j ACCEPT -s) "list-addrs" script can be found here[3] Note : * "iptables-restore" doesn't suffer of that netfilter regression, and I'm also aware that "iptables-restore" is the favourite approach since it is way more efficient than iptables that is executed over and over, once for each policy one want to set, but since "/sbin/iptables" takes vastly longer to perform with that commit, I think this need to be address anyway. * I also tried with the latest and greatest iptables upstream code, and got the same result. Reference : [1] - https://github.com/torvalds/linux/commit/71ae0dff02d756e4d2ca710b79f2ff5390029a5f [2] - https://gist.github.com/williammartin/b75e3faf5964648299e4d985413e6c0c [3] - https://gist.github.com/williammartin/b75e3faf5964648299e4d985413e6c0c#file-list-addrs [Impact] It has been brought to my attention that Ubuntu kernel 4.4 has a severe netfilter regression affecting the performance of "/sbin/iptables" command, especially when adding large number of policies. My source have documented everything here[2]. I was able to reproduce the situation on my side, and a kernel bisect identified the same offending commit[1] as my source found for this bug. Running the commit right before the offending one have proven to have expected performance : # commit [71ae0dff] <== Offending commit real 0m33.314s user 0m1.520s sys 0m26.192s # commit [d7b59742] <== Right before offending commit real 0m5.952s user 0m0.124s sys 0m0.220s [Test Case] * Reproducer #1 $ iptables -F $ time (./list-addrs 3000 | xargs -n1 iptables -A FORWARD -j ACCEPT -s) * Reproducer #2 $ iptables -F $ time for f in `seq 1 2000` ; do iptables -A FORWARD ; done "list-addrs" script can be found here[3] [Regression Potential] * none expected, the patches have been proven to work on mainline kernel, and was reviewed by a few netfilters maintainer + tested by myself. Reference: https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/ Patches: https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/2394ae21e8b652aff0db1c02e946243c1e2f5edb https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/722d6785e3b29a3b9f95c4d77542a1416094786a https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/18b61e8161cc308cbfd06d2e2c6c0758dfd925ef [Other Info] * "iptables-restore" doesn't suffer of that netfilter regression, and I'm also aware that "iptables-restore" is the favourite approach since it is way more efficient than iptables that is executed over and over, once for each policy one want to set, but since "binary arp/ip/ip6tables" takes vastly longer to perform with that commit, I think this need to be address anyway. [Related Documents] [1] - https://github.com/torvalds/linux/commit/71ae0dff02d756e4d2ca710b79f2ff5390029a5f [2] - https://gist.github.com/williammartin/b75e3faf5964648299e4d985413e6c0c [3] - https://gist.github.com/williammartin/b75e3faf5964648299e4d985413e6c0c#file-list-addrs
2016-12-06 20:08:12 Eric Desrochers description [Impact] It has been brought to my attention that Ubuntu kernel 4.4 has a severe netfilter regression affecting the performance of "/sbin/iptables" command, especially when adding large number of policies. My source have documented everything here[2]. I was able to reproduce the situation on my side, and a kernel bisect identified the same offending commit[1] as my source found for this bug. Running the commit right before the offending one have proven to have expected performance : # commit [71ae0dff] <== Offending commit real 0m33.314s user 0m1.520s sys 0m26.192s # commit [d7b59742] <== Right before offending commit real 0m5.952s user 0m0.124s sys 0m0.220s [Test Case] * Reproducer #1 $ iptables -F $ time (./list-addrs 3000 | xargs -n1 iptables -A FORWARD -j ACCEPT -s) * Reproducer #2 $ iptables -F $ time for f in `seq 1 2000` ; do iptables -A FORWARD ; done "list-addrs" script can be found here[3] [Regression Potential] * none expected, the patches have been proven to work on mainline kernel, and was reviewed by a few netfilters maintainer + tested by myself. Reference: https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/ Patches: https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/2394ae21e8b652aff0db1c02e946243c1e2f5edb https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/722d6785e3b29a3b9f95c4d77542a1416094786a https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/18b61e8161cc308cbfd06d2e2c6c0758dfd925ef [Other Info] * "iptables-restore" doesn't suffer of that netfilter regression, and I'm also aware that "iptables-restore" is the favourite approach since it is way more efficient than iptables that is executed over and over, once for each policy one want to set, but since "binary arp/ip/ip6tables" takes vastly longer to perform with that commit, I think this need to be address anyway. [Related Documents] [1] - https://github.com/torvalds/linux/commit/71ae0dff02d756e4d2ca710b79f2ff5390029a5f [2] - https://gist.github.com/williammartin/b75e3faf5964648299e4d985413e6c0c [3] - https://gist.github.com/williammartin/b75e3faf5964648299e4d985413e6c0c#file-list-addrs [SRU JUSTIFICATION] [Impact] It has been brought to my attention that Ubuntu kernel 4.4 has a severe netfilter regression affecting the performance of "/sbin/iptables" command, especially when adding large number of policies. My source have documented everything here[2]. I was able to reproduce the situation on my side, and a kernel bisect identified the same offending commit[1] as my source found for this bug. Running the commit right before the offending one have proven to have expected performance : # commit [71ae0dff] <== Offending commit real 0m33.314s user 0m1.520s sys 0m26.192s # commit [d7b59742] <== Right before offending commit real 0m5.952s user 0m0.124s sys 0m0.220s [Test Case] * Reproducer #1 $ iptables -F $ time (./list-addrs 3000 | xargs -n1 iptables -A FORWARD -j ACCEPT -s) * Reproducer #2 $ iptables -F $ time for f in `seq 1 2000` ; do iptables -A FORWARD ; done "list-addrs" script can be found here[3] [Regression Potential]  * none expected, the patches have been proven to work on mainline kernel, and was reviewed by a few netfilters maintainer + tested by myself. Reference: https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/ Patches: https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/2394ae21e8b652aff0db1c02e946243c1e2f5edb https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/722d6785e3b29a3b9f95c4d77542a1416094786a https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/18b61e8161cc308cbfd06d2e2c6c0758dfd925ef [Other Info] * "iptables-restore" doesn't suffer of that netfilter regression, and I'm also aware that "iptables-restore" is the favourite approach since it is way more efficient than iptables that is executed over and over, once for each policy one want to set, but since "binary arp/ip/ip6tables" takes vastly longer to perform with that commit, I think this need to be address anyway. [Related Documents] [1] - https://github.com/torvalds/linux/commit/71ae0dff02d756e4d2ca710b79f2ff5390029a5f [2] - https://gist.github.com/williammartin/b75e3faf5964648299e4d985413e6c0c [3] - https://gist.github.com/williammartin/b75e3faf5964648299e4d985413e6c0c#file-list-addrs
2016-12-06 20:11:31 Eric Desrochers description [SRU JUSTIFICATION] [Impact] It has been brought to my attention that Ubuntu kernel 4.4 has a severe netfilter regression affecting the performance of "/sbin/iptables" command, especially when adding large number of policies. My source have documented everything here[2]. I was able to reproduce the situation on my side, and a kernel bisect identified the same offending commit[1] as my source found for this bug. Running the commit right before the offending one have proven to have expected performance : # commit [71ae0dff] <== Offending commit real 0m33.314s user 0m1.520s sys 0m26.192s # commit [d7b59742] <== Right before offending commit real 0m5.952s user 0m0.124s sys 0m0.220s [Test Case] * Reproducer #1 $ iptables -F $ time (./list-addrs 3000 | xargs -n1 iptables -A FORWARD -j ACCEPT -s) * Reproducer #2 $ iptables -F $ time for f in `seq 1 2000` ; do iptables -A FORWARD ; done "list-addrs" script can be found here[3] [Regression Potential]  * none expected, the patches have been proven to work on mainline kernel, and was reviewed by a few netfilters maintainer + tested by myself. Reference: https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/ Patches: https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/2394ae21e8b652aff0db1c02e946243c1e2f5edb https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/722d6785e3b29a3b9f95c4d77542a1416094786a https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/18b61e8161cc308cbfd06d2e2c6c0758dfd925ef [Other Info] * "iptables-restore" doesn't suffer of that netfilter regression, and I'm also aware that "iptables-restore" is the favourite approach since it is way more efficient than iptables that is executed over and over, once for each policy one want to set, but since "binary arp/ip/ip6tables" takes vastly longer to perform with that commit, I think this need to be address anyway. [Related Documents] [1] - https://github.com/torvalds/linux/commit/71ae0dff02d756e4d2ca710b79f2ff5390029a5f [2] - https://gist.github.com/williammartin/b75e3faf5964648299e4d985413e6c0c [3] - https://gist.github.com/williammartin/b75e3faf5964648299e4d985413e6c0c#file-list-addrs [SRU JUSTIFICATION] [Impact] It has been brought to my attention that Ubuntu kernel 4.4 has a severe netfilter regression affecting the performance of "/sbin/iptables" command, especially when adding large number of policies. My source have documented everything here[2]. Note that the situation can also be reproduce with latest and greatest upstream kernel v4.9-rc4. I was able to reproduce the situation on my side, and a kernel bisect identified the same offending commit[1] as my source found for this bug. Running the commit right before the offending one have proven to have expected performance : # commit [71ae0dff] <== Offending commit real 0m33.314s user 0m1.520s sys 0m26.192s # commit [d7b59742] <== Right before offending commit real 0m5.952s user 0m0.124s sys 0m0.220s [Test Case] * Reproducer #1 $ iptables -F $ time (./list-addrs 3000 | xargs -n1 iptables -A FORWARD -j ACCEPT -s) * Reproducer #2 $ iptables -F $ time for f in `seq 1 2000` ; do iptables -A FORWARD ; done "list-addrs" script can be found here[3] [Regression Potential]  * none expected, the patches have been proven to work on mainline kernel, and was reviewed by a few netfilters maintainer + tested by myself. Reference: https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/ Patches: https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/2394ae21e8b652aff0db1c02e946243c1e2f5edb https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/722d6785e3b29a3b9f95c4d77542a1416094786a https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/18b61e8161cc308cbfd06d2e2c6c0758dfd925ef [Other Info] * "iptables-restore" doesn't suffer of that netfilter regression, and I'm also aware that "iptables-restore" is the favourite approach since it is way more efficient than iptables that is executed over and over, once for each policy one want to set, but since "binary arp/ip/ip6tables" takes vastly longer to perform with that commit, I think this need to be address anyway. [Related Documents] [1] - https://github.com/torvalds/linux/commit/71ae0dff02d756e4d2ca710b79f2ff5390029a5f [2] - https://gist.github.com/williammartin/b75e3faf5964648299e4d985413e6c0c [3] - https://gist.github.com/williammartin/b75e3faf5964648299e4d985413e6c0c#file-list-addrs
2016-12-06 20:13:35 Eric Desrochers description [SRU JUSTIFICATION] [Impact] It has been brought to my attention that Ubuntu kernel 4.4 has a severe netfilter regression affecting the performance of "/sbin/iptables" command, especially when adding large number of policies. My source have documented everything here[2]. Note that the situation can also be reproduce with latest and greatest upstream kernel v4.9-rc4. I was able to reproduce the situation on my side, and a kernel bisect identified the same offending commit[1] as my source found for this bug. Running the commit right before the offending one have proven to have expected performance : # commit [71ae0dff] <== Offending commit real 0m33.314s user 0m1.520s sys 0m26.192s # commit [d7b59742] <== Right before offending commit real 0m5.952s user 0m0.124s sys 0m0.220s [Test Case] * Reproducer #1 $ iptables -F $ time (./list-addrs 3000 | xargs -n1 iptables -A FORWARD -j ACCEPT -s) * Reproducer #2 $ iptables -F $ time for f in `seq 1 2000` ; do iptables -A FORWARD ; done "list-addrs" script can be found here[3] [Regression Potential]  * none expected, the patches have been proven to work on mainline kernel, and was reviewed by a few netfilters maintainer + tested by myself. Reference: https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/ Patches: https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/2394ae21e8b652aff0db1c02e946243c1e2f5edb https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/722d6785e3b29a3b9f95c4d77542a1416094786a https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/18b61e8161cc308cbfd06d2e2c6c0758dfd925ef [Other Info] * "iptables-restore" doesn't suffer of that netfilter regression, and I'm also aware that "iptables-restore" is the favourite approach since it is way more efficient than iptables that is executed over and over, once for each policy one want to set, but since "binary arp/ip/ip6tables" takes vastly longer to perform with that commit, I think this need to be address anyway. [Related Documents] [1] - https://github.com/torvalds/linux/commit/71ae0dff02d756e4d2ca710b79f2ff5390029a5f [2] - https://gist.github.com/williammartin/b75e3faf5964648299e4d985413e6c0c [3] - https://gist.github.com/williammartin/b75e3faf5964648299e4d985413e6c0c#file-list-addrs [SRU JUSTIFICATION] [Impact] It has been brought to my attention that Ubuntu kernel 4.4 has a severe netfilter regression affecting the performance of "/sbin/iptables" command, especially when adding large number of policies. My source have documented everything here[2]. Note that the situation can also be reproduce with latest and greatest upstream kernel v4.9-rc4. I was able to reproduce the situation on my side, and a kernel bisect identified the same offending commit[1] as my source found for this bug. Running the commit right before the offending one have proven to have expected performance : # commit [71ae0dff] <== Offending commit real 0m33.314s user 0m1.520s sys 0m26.192s # commit [d7b59742] <== Right before offending commit real 0m5.952s user 0m0.124s sys 0m0.220s [Test Case] * Reproducer #1 $ iptables -F $ time (./list-addrs 3000 | xargs -n1 iptables -A FORWARD -j ACCEPT -s) * Reproducer #2 $ iptables -F $ time for f in `seq 1 3000` ; do iptables -A FORWARD ; done "list-addrs" script can be found here[3] [Regression Potential]  * none expected, the patches have been proven to work on mainline kernel, and was reviewed by a few netfilters maintainer + tested by myself. Reference: https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/ Patches: https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/2394ae21e8b652aff0db1c02e946243c1e2f5edb https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/722d6785e3b29a3b9f95c4d77542a1416094786a https://kernel.googlesource.com/pub/scm/linux/kernel/git/pablo/nf-next/+/18b61e8161cc308cbfd06d2e2c6c0758dfd925ef [Other Info] * "iptables-restore" doesn't suffer of that netfilter regression, and I'm also aware that "iptables-restore" is the favourite approach since it is way more efficient than iptables that is executed over and over, once for each policy one want to set, but since "binary arp/ip/ip6tables" takes vastly longer to perform with that commit, I think this need to be address anyway. [Related Documents] [1] - https://github.com/torvalds/linux/commit/71ae0dff02d756e4d2ca710b79f2ff5390029a5f [2] - https://gist.github.com/williammartin/b75e3faf5964648299e4d985413e6c0c [3] - https://gist.github.com/williammartin/b75e3faf5964648299e4d985413e6c0c#file-list-addrs
2016-12-07 13:18:55 Robie Basak nominated for series Ubuntu Yakkety
2016-12-07 13:18:55 Robie Basak bug task added linux (Ubuntu Yakkety)
2016-12-07 13:19:49 Eric Desrochers linux (Ubuntu Yakkety): importance Undecided High
2016-12-07 13:19:52 Eric Desrochers linux (Ubuntu Yakkety): assignee Eric Desrochers (slashd)
2016-12-07 13:19:55 Eric Desrochers linux (Ubuntu Yakkety): status New In Progress
2017-01-20 12:17:12 Luis Henriques linux (Ubuntu Yakkety): status In Progress Fix Committed
2017-01-20 12:19:37 Luis Henriques linux (Ubuntu Xenial): status In Progress Fix Committed
2017-01-31 22:39:38 Launchpad Janitor linux (Ubuntu): status In Progress Fix Released
2017-02-09 22:02:40 Thadeu Lima de Souza Cascardo tags kernel-da-key sts kernel-da-key sts verification-needed-xenial
2017-02-09 22:03:26 Thadeu Lima de Souza Cascardo tags kernel-da-key sts verification-needed-xenial kernel-da-key sts verification-needed-xenial verification-needed-yakkety
2017-02-09 22:22:33 Eric Desrochers tags kernel-da-key sts verification-needed-xenial verification-needed-yakkety kernel-da-key sts verification-done-xenial verification-needed-yakkety
2017-02-13 14:48:26 Eric Desrochers tags kernel-da-key sts verification-done-xenial verification-needed-yakkety kernel-da-key sts verification-done-xenial verification-done-yakkety
2017-02-20 07:42:07 Launchpad Janitor linux (Ubuntu Xenial): status Fix Committed Fix Released
2017-02-20 07:46:10 Launchpad Janitor linux (Ubuntu Yakkety): status Fix Committed Fix Released