sssd appears to crash AWS c5 and m5 instances, cause 100% CPU
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
cloud-images |
Fix Released
|
High
|
Unassigned | ||
linux (Ubuntu) |
In Progress
|
Critical
|
Kamal Mostafa | ||
Xenial |
In Progress
|
Undecided
|
Kamal Mostafa | ||
linux-aws (Ubuntu) |
Confirmed
|
Critical
|
Kamal Mostafa | ||
Xenial |
In Progress
|
Undecided
|
Kamal Mostafa |
Bug Description
After upgrading to the Ubuntu EC2 AMI from 20180126 (specifically ami-79873901 in us-west-2) we have seen sssd hard locking c5 and m5 EC2 instances after starting the service and CPU goes to 100%.
We do not experience this issue with t2 or c4 instance types and we do not see this issue on any instance types using Ubuntu Cloud images from 20180109 or before. I have verified that this is kernel related as I booted an image that we created using the Ubuntu cloud image from 20180109 which works fine on a c5. I then did a "apt update && apt install --only-upgrade linux-aws && systemctl disable sssd", rebooted the server, verified I was on the new kernel and started sssd with "systemctl start sssd" and the EC2 instance froze and Cloudwatch CPU usage for that instance went to 100%.
I haven't been able to find much in the syslog, kern.log, journalctl logs, etc. The only thing I have been able to find is that when this happens I tend to see "^@^@^@
Thanks,
Paul
Changed in linux (Ubuntu): | |
importance: | Undecided → Critical |
tags: | added: kernel-key |
tags: | added: pti |
Changed in linux (Ubuntu): | |
assignee: | Joseph Salisbury (jsalisbury) → Kamal Mostafa (kamalmostafa) |
Changed in linux-aws (Ubuntu): | |
assignee: | Joseph Salisbury (jsalisbury) → Kamal Mostafa (kamalmostafa) |
no longer affects: | sssd (Ubuntu) |
no longer affects: | sssd (Ubuntu Xenial) |
Changed in linux-aws (Ubuntu Xenial): | |
assignee: | nobody → Kamal Mostafa (kamalmostafa) |
status: | New → In Progress |
Changed in linux (Ubuntu Xenial): | |
assignee: | nobody → Kamal Mostafa (kamalmostafa) |
status: | New → In Progress |
Changed in linux (Ubuntu Xenial): | |
status: | In Progress → Fix Committed |
Changed in linux-aws (Ubuntu Xenial): | |
status: | In Progress → Fix Committed |
tags: | removed: kernel-key pti |
Changed in cloud-images: | |
status: | New → In Progress |
importance: | Undecided → High |
Changed in cloud-images: | |
status: | In Progress → Fix Released |
Changed in linux (Ubuntu): | |
status: | Confirmed → In Progress |
Changed in linux (Ubuntu Xenial): | |
status: | Confirmed → In Progress |
Changed in linux-aws (Ubuntu): | |
status: | Confirmed → In Progress |
Changed in linux-aws (Ubuntu Xenial): | |
status: | Confirmed → In Progress |
Changed in linux-aws (Ubuntu): | |
status: | In Progress → Confirmed |
Added the cloud-images project so we can more easily track this.