kernels compiled with"CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y" lead to weird clock drifts on some CPUs. This may up ending in DOS
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux-ec2 (Ubuntu) |
New
|
Undecided
|
Unassigned | ||
linux-meta-ec2 (Ubuntu) |
Invalid
|
Undecided
|
Unassigned |
Bug Description
AWS research information added:
We have been investigating this issue. It is important to note that the issue here is not the wall clock, which is the source for gettimeofday(), etc. We're seeing differing behavior in the scheduler clock, which is responsible for process accounting. NTP is not a solution to this problem.
The large jump in the kernel printk() time stamp (as seen in dmesg) is due to the kernel reading the TSC (Time Stamp Counter) from the underlying CPU. The RDTSC instruction is not virtualized, so the counter will not start at zero.
Seeing the printk() timestamps jump around is an indication that the kernel is trusting TSC values too much for the scheduler clock. Most kernels are compiled with the "CONFIG_
Unfortunately, when the kernel detects some newer Intel processor models, it assumes that the hardware time source is stable enough to skip the mixing step. The code that makes the assumption is under recent discussion here.
From arch/x86/
if (c->x86_power & (1 << 8)) {
set_cpu_cap(c, X86_FEATURE_
set_cpu_cap(c, X86_FEATURE_
if (!check_
sched_
I built a test kernel with the code that forces sched_clock_
If you are working with a Linux vendor on this problem, point them at this code.
I hope this helps resolve your problem, and apologize that you encountered it while running on EC2.
Own comment:
For example this happens with "Intel(R) Xeon(R) CPU E5506 @ 2.13GHz", but not with "Intel(R) Xeon(R) CPU E5410 @ 2.33GHz". This makes EC2 Autoscaling kind of unreliable :(
ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: linux-ec2 2.6.32.311.12
ProcVersionSign
Uname: Linux 2.6.32-308-ec2 x86_64
Architecture: amd64
Date: Wed Mar 23 02:11:07 2011
Ec2AMI: ami-4583b731
Ec2AMIManifest: (unknown)
Ec2Availability
Ec2InstanceType: c1.xlarge
Ec2Kernel: aki-4e794c3a
Ec2Ramdisk: unavailable
ProcEnviron:
LANG=en_GB.UTF-8
SHELL=/bin/bash
SourcePackage: linux-meta-ec2
Sorry, forgot to add the AWS discussion thread: https:/ /forums. aws.amazon. com/thread. jspa?threadID= 59753