Ksmd eats I/O on host(lab)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Invalid
|
High
|
Alexander Arzhanov | ||
7.0.x |
Invalid
|
High
|
Alexander Arzhanov | ||
8.0.x |
Invalid
|
High
|
Alexander Arzhanov |
Bug Description
Problem description
At least two of our labs, we faced a problem with ksmd, ksmd constantly high disk I/O uses at host(lab).
It is reproduced in such a scenario:
1. Create cluster with vCenter support.
2. Add 3 nodes with Controller roles.
3. Add 2 nodes with compute role.
4. Deploy the cluster.
5. Run network verification.
6. Run OSTF.
7. Create 2 VMs on each of hypervisor.
8. Verify that VMs on different hypervisors should communicate
between each other.
In some cases, it took leave env with working instances (5-10 items) for some time (eg at night).
Likely, the problem arises in env without vcenter, specially we have not tested.
As the problem appears on the guest side:
1.At nodes observed jumps load average.
2.If you run ping node-1 -> node-2(for example), you can see high latency, sometimes packets will be lost.
3.Galera Cluster temporarily loses nodes (because of item 2).
4.Corosync occasionally loses nodes (because of item 2).
As the problem appears on the host side:
If you run iotop(for example), you can see, that ksmd using 40-80-99% (usually 99.9%) I/O on host(lab).
If you run echo 2 > /sys/kernel/
Some details about the labs:
Ubuntu 14.04.3 LTS
Kernel: 3.13.0-24
Libvirt 1.2.2
Solution
1.Update the kernel, there is no problem with the following kernel (the other was not tested) :
3.13.0-39
3.13.0-63
2.Perhaps the problem will solve update libvirt.
3.Do not use KSM.
Changed in fuel: | |
assignee: | nobody → MOS Linux (mos-linux) |
Changed in fuel: | |
importance: | Undecided → High |
status: | New → Confirmed |
tags: | added: area-partners |
Changed in fuel: | |
milestone: | 7.0-updates → 8.0 |
Looks linked: /bugs.launchpad .net/ubuntu/ +source/ linux/+ bug/1346917
https:/