Activity log for bug #1020285

Date Who What changed Old value New value Message
2012-07-02 20:39:44 James Troup bug added bug
2012-07-02 20:39:54 James Troup bug added subscriber The Canonical Sysadmins
2012-07-02 20:49:59 Joseph Salisbury linux (Ubuntu): importance Undecided Medium
2012-07-02 20:50:27 Joseph Salisbury tags kernel-da-key lucid precise
2012-07-02 20:58:11 Leann Ogasawara linux (Ubuntu): importance Medium Undecided
2012-07-02 20:58:11 Leann Ogasawara linux (Ubuntu): status New Triaged
2012-07-02 20:58:11 Leann Ogasawara linux (Ubuntu): assignee Canonical Kernel Team (canonical-kernel-team)
2012-07-02 20:58:22 Leann Ogasawara linux (Ubuntu): importance Undecided Medium
2012-07-02 21:35:31 Mark Russell bug added subscriber Canonical Support
2012-07-03 00:04:20 Nobuto Murata bug added subscriber Nobuto MURATA
2012-07-03 00:20:37 abiru bug added subscriber abiru
2012-07-03 00:36:25 Haw Loeung bug added subscriber Canonical WebOps
2012-07-03 02:05:32 Steve Langasek nominated for series Ubuntu Lucid
2012-07-03 02:05:32 Steve Langasek bug task added linux (Ubuntu Lucid)
2012-07-03 02:05:32 Steve Langasek nominated for series Ubuntu Natty
2012-07-03 02:05:32 Steve Langasek bug task added linux (Ubuntu Natty)
2012-07-03 02:05:32 Steve Langasek nominated for series Ubuntu Oneiric
2012-07-03 02:05:32 Steve Langasek bug task added linux (Ubuntu Oneiric)
2012-07-03 02:05:32 Steve Langasek nominated for series Ubuntu Precise
2012-07-03 02:05:32 Steve Langasek bug task added linux (Ubuntu Precise)
2012-07-03 02:05:32 Steve Langasek nominated for series Ubuntu Quantal
2012-07-03 02:05:32 Steve Langasek bug task added linux (Ubuntu Quantal)
2012-07-03 07:58:46 Robie Basak bug added subscriber Robie Basak
2012-07-03 09:50:16 Andy Whitcroft linux (Ubuntu Precise): importance Undecided Medium
2012-07-03 09:50:20 Andy Whitcroft linux (Ubuntu Oneiric): importance Undecided Medium
2012-07-03 09:50:22 Andy Whitcroft linux (Ubuntu Natty): importance Undecided Medium
2012-07-03 09:50:25 Andy Whitcroft linux (Ubuntu Lucid): importance Undecided Medium
2012-07-03 09:50:29 Andy Whitcroft linux (Ubuntu Quantal): assignee Canonical Kernel Team (canonical-kernel-team) Andy Whitcroft (apw)
2012-07-03 11:37:19 Andy Whitcroft linux (Ubuntu Quantal): assignee Andy Whitcroft (apw) Brad Figg (brad-figg)
2012-07-03 11:37:28 Andy Whitcroft linux (Ubuntu Quantal): status Triaged In Progress
2012-07-03 13:44:10 Joseph Salisbury linux (Ubuntu Precise): status New Confirmed
2012-07-03 13:44:15 Joseph Salisbury linux (Ubuntu Precise): status Confirmed Triaged
2012-07-03 13:44:31 Joseph Salisbury linux (Ubuntu Oneiric): status New Triaged
2012-07-03 13:44:34 Joseph Salisbury linux (Ubuntu Natty): status New Triaged
2012-07-03 13:44:36 Joseph Salisbury linux (Ubuntu Lucid): status New Triaged
2012-07-03 13:44:48 Joseph Salisbury tags kernel-da-key lucid precise kernel-da-key kernel-key lucid precise
2012-07-03 16:13:01 Brad Figg linux (Ubuntu Lucid): assignee Brad Figg (brad-figg)
2012-07-03 16:13:05 Brad Figg linux (Ubuntu Natty): assignee Brad Figg (brad-figg)
2012-07-03 16:13:08 Brad Figg linux (Ubuntu Oneiric): assignee Brad Figg (brad-figg)
2012-07-03 16:13:10 Brad Figg linux (Ubuntu Precise): assignee Brad Figg (brad-figg)
2012-07-03 17:05:11 Steve Langasek bug task added base-files (Ubuntu)
2012-07-03 17:45:18 Launchpad Janitor branch linked lp:ubuntu/base-files
2012-07-03 18:00:13 Launchpad Janitor base-files (Ubuntu Quantal): status New Fix Released
2012-07-03 18:25:30 Adam Conrad base-files (Ubuntu Precise): status New Fix Committed
2012-07-03 18:25:33 Adam Conrad linux (Ubuntu Precise): status Triaged Fix Committed
2012-07-03 18:25:38 Adam Conrad bug added subscriber Ubuntu Stable Release Updates Team
2012-07-03 18:25:43 Adam Conrad bug added subscriber SRU Verification
2012-07-03 18:26:30 Adam Conrad linux (Ubuntu Precise): status Fix Committed New
2012-07-03 18:26:43 Adam Conrad linux (Ubuntu Precise): status New Triaged
2012-07-03 18:29:03 Adam Conrad base-files (Ubuntu Oneiric): status New Fix Committed
2012-07-03 18:29:10 Adam Conrad tags kernel-da-key kernel-key lucid precise kernel-da-key kernel-key lucid precise verification-needed
2012-07-03 18:34:45 Steve Langasek description As widely reported, the addition of the leap second on 2012-06-30 has caused high CPU usage and futex lockups in a lot of applications including JVMs, Mysql as well as desktop apps like Firefox and Thunderbird. https://lkml.org/lkml/2012/6/30/122 http://serverfault.com/questions/403732/anyone-else-experiencing-high-rates-of-linux-server-crashes-during-a-leap-second https://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/ We've seen this ourselves on the Canonical infrastructure on both current Lucid and Precise kernels, i.e. ii linux-image-2.6.32-41-server 2.6.32-41.90 Linux kernel image for version 2.6.32 on x86_64 ii linux-image-3.2.0-24-generic 3.2.0-24.39 Linux kernel image for version 3.2.0 on 64 bit x86 SMP We can also confirm the 'date -s $(date)' workaround fixes the problem without requiring a reboot. [Impact] Software that relies on fine-grained pthread timeouts will spin indefinitely and drive up system load following a leap second, when the kernel's idea of time has become desynced and sub-1s timeouts are all hit immediately. Mysql and Java are in particular reported to be affected by this. This is a transient issue, in that it will go away the first time the system is rebooted after the leap second and is expected to be fixed before the next leap second occurs; nevertheless admins have been caught off-guard by this misbehavior and in some cases may not have noticed the problem or know what to do about it, so we should help them along by resetting the kernel clock with a minimal-risk base-files update. [Test Case] 1. Find a system that has been online, with mysqld or a java-based process running since before 2012-06-30. 2. Verify that one or more processes on the system are spinning in futex and driving up the system load. 3. Upgrade to the base-files package from -proposed. 4. Verify that the system load comes back down immediately. [Regression potential] No analysis has been done on the effect of resetting the date on applications that require a high-accuracy clock. While this fixes the problem with the pthreads interfaces, it may cause other problems for other software. Since the proposed fix is to reset the kernel's date to the current date, which is not atomic, there will be a slight skew of the clock backwards in time. ntp *should* fix this shortly thereafter for machines that have it enabled. As widely reported, the addition of the leap second on 2012-06-30 has caused high CPU usage and futex lockups in a lot of applications including JVMs, Mysql as well as desktop apps like Firefox and Thunderbird. https://lkml.org/lkml/2012/6/30/122 http://serverfault.com/questions/403732/anyone-else-experiencing-high-rates-of-linux-server-crashes-during-a-leap-second https://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/ We've seen this ourselves on the Canonical infrastructure on both current Lucid and Precise kernels, i.e. ii linux-image-2.6.32-41-server 2.6.32-41.90 Linux kernel image for version 2.6.32 on x86_64 ii linux-image-3.2.0-24-generic 3.2.0-24.39 Linux kernel image for version 3.2.0 on 64 bit x86 SMP We can also confirm the 'date -s $(date)' workaround fixes the problem without requiring a reboot.
2012-07-03 18:43:40 Adam Conrad description [Impact] Software that relies on fine-grained pthread timeouts will spin indefinitely and drive up system load following a leap second, when the kernel's idea of time has become desynced and sub-1s timeouts are all hit immediately. Mysql and Java are in particular reported to be affected by this. This is a transient issue, in that it will go away the first time the system is rebooted after the leap second and is expected to be fixed before the next leap second occurs; nevertheless admins have been caught off-guard by this misbehavior and in some cases may not have noticed the problem or know what to do about it, so we should help them along by resetting the kernel clock with a minimal-risk base-files update. [Test Case] 1. Find a system that has been online, with mysqld or a java-based process running since before 2012-06-30. 2. Verify that one or more processes on the system are spinning in futex and driving up the system load. 3. Upgrade to the base-files package from -proposed. 4. Verify that the system load comes back down immediately. [Regression potential] No analysis has been done on the effect of resetting the date on applications that require a high-accuracy clock. While this fixes the problem with the pthreads interfaces, it may cause other problems for other software. Since the proposed fix is to reset the kernel's date to the current date, which is not atomic, there will be a slight skew of the clock backwards in time. ntp *should* fix this shortly thereafter for machines that have it enabled. As widely reported, the addition of the leap second on 2012-06-30 has caused high CPU usage and futex lockups in a lot of applications including JVMs, Mysql as well as desktop apps like Firefox and Thunderbird. https://lkml.org/lkml/2012/6/30/122 http://serverfault.com/questions/403732/anyone-else-experiencing-high-rates-of-linux-server-crashes-during-a-leap-second https://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/ We've seen this ourselves on the Canonical infrastructure on both current Lucid and Precise kernels, i.e. ii linux-image-2.6.32-41-server 2.6.32-41.90 Linux kernel image for version 2.6.32 on x86_64 ii linux-image-3.2.0-24-generic 3.2.0-24.39 Linux kernel image for version 3.2.0 on 64 bit x86 SMP We can also confirm the 'date -s $(date)' workaround fixes the problem without requiring a reboot. [Impact] Software that relies on fine-grained pthread timeouts will spin indefinitely and drive up system load following a leap second, when the kernel's idea of time has become desynced and sub-1s timeouts are all hit immediately. Mysql and Java are in particular reported to be affected by this. This is a transient issue, in that it will go away the first time the system is rebooted after the leap second and is expected to be fixed before the next leap second occurs; nevertheless admins have been caught off-guard by this misbehavior and in some cases may not have noticed the problem or know what to do about it, so we should help them along by resetting the kernel clock with a minimal-risk base-files update. [Test Case] 1. Find a system that has been online, with mysqld or a java-based process running since before 2012-06-30. 2. Verify that one or more processes on the system are spinning in futex and driving up the system load. 3. Upgrade to the base-files package from -proposed. 4. Verify that the system load comes back down immediately. 5. A stress-test for leap-second handling has been provided at https://lkml.org/lkml/2012/7/3/37 [Regression potential] No analysis has been done on the effect of resetting the date on applications that require a high-accuracy clock. While this fixes the problem with the pthreads interfaces, it may cause other problems for other software. Since the proposed fix is to reset the kernel's date to the current date, which is not atomic, there will be a slight skew of the clock backwards in time. ntp *should* fix this shortly thereafter for machines that have it enabled. As widely reported, the addition of the leap second on 2012-06-30 has caused high CPU usage and futex lockups in a lot of applications including JVMs, Mysql as well as desktop apps like Firefox and Thunderbird. https://lkml.org/lkml/2012/6/30/122 http://serverfault.com/questions/403732/anyone-else-experiencing-high-rates-of-linux-server-crashes-during-a-leap-second https://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/ We've seen this ourselves on the Canonical infrastructure on both current Lucid and Precise kernels, i.e. ii linux-image-2.6.32-41-server 2.6.32-41.90 Linux kernel image for version 2.6.32 on x86_64 ii linux-image-3.2.0-24-generic 3.2.0-24.39 Linux kernel image for version 3.2.0 on 64 bit x86 SMP We can also confirm the 'date -s $(date)' workaround fixes the problem without requiring a reboot.
2012-07-03 18:43:56 Launchpad Janitor branch linked lp:~ubuntu-branches/ubuntu/oneiric/base-files/oneiric-proposed
2012-07-03 18:44:06 Launchpad Janitor branch linked lp:~ubuntu-branches/ubuntu/precise/base-files/precise-proposed
2012-07-03 18:44:57 Adam Conrad base-files (Ubuntu Natty): status New Fix Committed
2012-07-03 18:46:00 Steve Langasek description [Impact] Software that relies on fine-grained pthread timeouts will spin indefinitely and drive up system load following a leap second, when the kernel's idea of time has become desynced and sub-1s timeouts are all hit immediately. Mysql and Java are in particular reported to be affected by this. This is a transient issue, in that it will go away the first time the system is rebooted after the leap second and is expected to be fixed before the next leap second occurs; nevertheless admins have been caught off-guard by this misbehavior and in some cases may not have noticed the problem or know what to do about it, so we should help them along by resetting the kernel clock with a minimal-risk base-files update. [Test Case] 1. Find a system that has been online, with mysqld or a java-based process running since before 2012-06-30. 2. Verify that one or more processes on the system are spinning in futex and driving up the system load. 3. Upgrade to the base-files package from -proposed. 4. Verify that the system load comes back down immediately. 5. A stress-test for leap-second handling has been provided at https://lkml.org/lkml/2012/7/3/37 [Regression potential] No analysis has been done on the effect of resetting the date on applications that require a high-accuracy clock. While this fixes the problem with the pthreads interfaces, it may cause other problems for other software. Since the proposed fix is to reset the kernel's date to the current date, which is not atomic, there will be a slight skew of the clock backwards in time. ntp *should* fix this shortly thereafter for machines that have it enabled. As widely reported, the addition of the leap second on 2012-06-30 has caused high CPU usage and futex lockups in a lot of applications including JVMs, Mysql as well as desktop apps like Firefox and Thunderbird. https://lkml.org/lkml/2012/6/30/122 http://serverfault.com/questions/403732/anyone-else-experiencing-high-rates-of-linux-server-crashes-during-a-leap-second https://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/ We've seen this ourselves on the Canonical infrastructure on both current Lucid and Precise kernels, i.e. ii linux-image-2.6.32-41-server 2.6.32-41.90 Linux kernel image for version 2.6.32 on x86_64 ii linux-image-3.2.0-24-generic 3.2.0-24.39 Linux kernel image for version 3.2.0 on 64 bit x86 SMP We can also confirm the 'date -s $(date)' workaround fixes the problem without requiring a reboot. [Impact] Software that relies on fine-grained pthread timeouts will spin indefinitely and drive up system load following a leap second, when the kernel's idea of time has become desynced and sub-1s timeouts are all hit immediately. Mysql and Java are in particular reported to be affected by this. This is a transient issue, in that it will go away the first time the system is rebooted after the leap second and is expected to be fixed before the next leap second occurs; nevertheless admins have been caught off-guard by this misbehavior and in some cases may not have noticed the problem or know what to do about it, so we should help them along by resetting the kernel clock with a minimal-risk base-files update. [Test Case] 1. Find a system that has been online, with mysqld or a java-based process running since before 2012-06-30. 2. Verify that one or more processes on the system are spinning in futex and driving up the system load. 3. Upgrade to the base-files package from -proposed. 4. Verify that the system load comes back down immediately. 5. A stress-test for leap-second handling has been provided at https://lkml.org/lkml/2012/7/3/37 [Regression potential] No analysis has been done on the effect of resetting the date on applications that require a high-accuracy clock. While this fixes the problem with the pthreads interfaces, it may cause other problems for other software. Since the proposed fix is to reset the kernel's date to the current date, which is not atomic, there will be a slight skew of the clock backwards in time. ntp *should* fix this shortly thereafter for machines that have it enabled. Also, because there's a single version check for each copy of the SRU, users whose applications are negatively affected by the running of this date command will also be negatively affected on each subsequent upgrade of the system, up to and including the quantal devel release. As widely reported, the addition of the leap second on 2012-06-30 has caused high CPU usage and futex lockups in a lot of applications including JVMs, Mysql as well as desktop apps like Firefox and Thunderbird. https://lkml.org/lkml/2012/6/30/122 http://serverfault.com/questions/403732/anyone-else-experiencing-high-rates-of-linux-server-crashes-during-a-leap-second https://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/ We've seen this ourselves on the Canonical infrastructure on both current Lucid and Precise kernels, i.e. ii linux-image-2.6.32-41-server 2.6.32-41.90 Linux kernel image for version 2.6.32 on x86_64 ii linux-image-3.2.0-24-generic 3.2.0-24.39 Linux kernel image for version 3.2.0 on 64 bit x86 SMP We can also confirm the 'date -s $(date)' workaround fixes the problem without requiring a reboot.
2012-07-03 18:54:46 Adam Conrad base-files (Ubuntu Lucid): status New Fix Committed
2012-07-03 19:04:50 Launchpad Janitor branch linked lp:~ubuntu-branches/ubuntu/natty/base-files/natty-proposed
2012-07-03 19:17:26 Launchpad Janitor branch linked lp:ubuntu/lucid-proposed/base-files
2012-07-05 19:30:32 Alex Gorban bug added subscriber Alex Gorban
2012-07-06 08:49:26 Itaru Kitayama bug added subscriber Itaru Kitayama
2012-07-06 11:02:31 Dave Russell bug added subscriber Dave Russell
2012-07-06 22:58:24 Nathan Stratton Treadway bug added subscriber Nathan Stratton Treadway
2012-07-07 19:45:14 Nathan Stratton Treadway bug watch added http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=679882
2012-07-09 20:47:08 Steve Langasek base-files (Ubuntu Lucid): status Fix Committed Won't Fix
2012-07-09 20:47:32 Steve Langasek base-files (Ubuntu Natty): status Fix Committed Won't Fix
2012-07-09 20:48:17 Steve Langasek base-files (Ubuntu Oneiric): status Fix Committed Won't Fix
2012-07-09 20:48:44 Steve Langasek base-files (Ubuntu Precise): status Fix Committed Won't Fix
2012-07-09 20:49:13 Steve Langasek base-files (Ubuntu Quantal): status Fix Released Won't Fix
2012-07-12 04:20:50 Nathan Stratton Treadway attachment added Python script to check for post-leap-second sub-second timeouts issue https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1020285/+attachment/3220508/+files/post_leapsecond_nanosleep_check.py
2012-07-12 04:56:57 Nathan Stratton Treadway attachment removed Python script to check for post-leap-second sub-second timeouts issue https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1020285/+attachment/3220508/+files/post_leapsecond_nanosleep_check.py
2012-07-12 05:01:01 Nathan Stratton Treadway attachment added Python script to check for post-leap-second sub-second timeouts issue https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1020285/+attachment/3220568/+files/post_leapsecond_nanosleep_check.py
2012-07-12 22:09:50 Nathan Stratton Treadway bug watch added https://bugzilla.redhat.com/show_bug.cgi?id=836803
2012-07-24 15:24:22 Tim Gardner linux (Ubuntu Lucid): status Triaged Fix Committed
2012-07-27 20:27:23 Launchpad Janitor branch linked lp:ubuntu/lucid-proposed/linux-ec2
2012-07-30 09:46:18 Luis Henriques tags kernel-da-key kernel-key lucid precise verification-needed kernel-da-key kernel-key lucid precise verification-needed verification-needed-lucid
2012-07-30 10:31:07 Luis Henriques tags kernel-da-key kernel-key lucid precise verification-needed verification-needed-lucid kernel-da-key kernel-key lucid precise verification-done-lucid verification-needed
2012-08-01 17:45:09 Abel Lopez bug added subscriber Abel Lopez
2012-08-10 00:24:35 Adam Conrad removed subscriber Ubuntu Stable Release Updates Team
2012-08-10 00:25:46 Launchpad Janitor linux (Ubuntu Lucid): status Fix Committed Fix Released
2012-08-10 00:25:46 Launchpad Janitor cve linked 2012-2136
2012-08-10 00:25:46 Launchpad Janitor cve linked 2012-2390
2012-08-28 18:50:13 Joseph Salisbury tags kernel-da-key kernel-key lucid precise verification-done-lucid verification-needed kernel-da-key lucid precise verification-done-lucid verification-needed
2012-08-29 20:28:52 Herton R. Krzesinski linux (Ubuntu Oneiric): status Triaged Fix Released
2012-08-29 20:29:07 Herton R. Krzesinski linux (Ubuntu Precise): status Triaged Fix Released
2012-08-29 20:36:55 Herton R. Krzesinski linux (Ubuntu Quantal): status In Progress Fix Released
2012-09-22 10:42:45 jan bug added subscriber jan
2013-08-18 17:19:42 Julian Wiedmann linux (Ubuntu Natty): status Triaged Invalid