Application-isolated type CPUs can't be used by Kubernetes Pod

[changyi]

Brief Description
-----------------

The configured application-isolated type CPUs can't be used by Kubernetes Pods and CPU resources required by the Pods are still from application type CPUs.

Severity
--------
Major

Steps to Reproduce
------------------

[sysadmin@controller-0 ~(keystone_admin)]$ system host-lock controller-0
[sysadmin@controller-0 ~(keystone_admin)]$ system host-cpu-modify -f application-isolated -p0 10 controller-0
[sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-0

[sysadmin@controller-0 ~(keystone_admin)]$ system host-cpu-list controller-0
+--------------------------------------+-------+-----------+-------+--------+------------------------------------------+----------------------+
| uuid | log_c | processor | phy_c | thread | processor_model | assigned_function |
| | ore | | ore | | | |
+--------------------------------------+-------+-----------+-------+--------+------------------------------------------+----------------------+
| 66b89d48-19fa-42fd-b0c8-1f2ff24b836a | 0 | 0 | 0 | 0 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Platform |
| 1507b67b-21c9-4023-a50b-6f9d5db8d5aa | 1 | 0 | 1 | 0 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Platform |
| e58537c7-f899-4d9b-b51a-c8823ad90218 | 2 | 0 | 2 | 0 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application-isolated |
| 9af9f09f-ce52-41a4-bbc9-6bc4742e4907 | 3 | 0 | 3 | 0 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application-isolated |
| ad6c2bfb-f7ee-4692-84ad-a4c0978bc923 | 4 | 0 | 4 | 0 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application-isolated |
| 09a54f20-83e7-4785-b49d-c4e56a0d0e32 | 5 | 0 | 8 | 0 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application-isolated |
| afdabd2c-a3c8-4807-a444-217eb9bada37 | 6 | 0 | 9 | 0 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application-isolated |
| 8118d082-088e-4778-bdcc-4e5d87a54d1e | 7 | 0 | 10 | 0 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application-isolated |
| 2d1972e1-041b-4b00-928a-22bb28758c83 | 8 | 0 | 11 | 0 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application-isolated |
| 4bd8277f-a427-4dad-9606-f87af7649c42 | 9 | 0 | 12 | 0 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application-isolated |
| fddfac0d-c11b-49e5-aa94-11af5b2791af | 10 | 0 | 16 | 0 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application-isolated |
| 5c3c0105-4db1-4834-9ac9-3d3c137b628a | 11 | 0 | 17 | 0 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application-isolated |
| 65ea6b4f-2785-427f-b818-47c0c23df88a | 12 | 0 | 18 | 0 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application |
| c9a586cc-5826-4e98-ac28-a65200f6be89 | 13 | 0 | 19 | 0 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application |
| 6311f078-e1b4-40e1-ae91-5f0d4508d672 | 14 | 0 | 20 | 0 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application |
| 21f98611-5e04-40e9-9e8f-dc13547b38f8 | 15 | 0 | 24 | 0 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application |
| fbe24d87-d5ba-4cfd-b1ac-7c781d1dae99 | 16 | 0 | 25 | 0 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application |
| 98df3f25-fa35-4140-b7aa-73062ecdc60b | 17 | 0 | 26 | 0 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application |
| 31dc5de2-6f98-41cf-afab-9fe9c6e94cd7 | 18 | 0 | 27 | 0 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application |
| 8d0ae62e-a01f-4cb9-952a-8e2a3e7ac66a | 19 | 0 | 28 | 0 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application |
| dbe320e3-ec65-4cc9-8d17-0814a0228051 | 20 | 0 | 0 | 1 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Platform |
| cc13160d-7954-41de-9b05-ac53d2b15cbc | 21 | 0 | 1 | 1 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Platform |
| dea796fc-96f0-49f5-b9e6-a203c594bf54 | 22 | 0 | 2 | 1 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application-isolated |
| 958a71ab-a9a8-40e3-9f3e-c4e30e4c28d1 | 23 | 0 | 3 | 1 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application-isolated |
| 94d1bc1c-ccb0-4e33-b2f4-c659ea815458 | 24 | 0 | 4 | 1 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application-isolated |
| 88f53809-941b-4ce2-a922-4bbb3cdd7e5d | 25 | 0 | 8 | 1 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application-isolated |
| 8be6b290-0f49-4a2d-adb0-825d692e3a9b | 26 | 0 | 9 | 1 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application-isolated |
| 6c3e712a-2867-4c0c-880c-23be3ce97499 | 27 | 0 | 10 | 1 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application-isolated |
| ee9a118e-9493-4e16-85d9-cfff9b28cb4f | 28 | 0 | 11 | 1 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application-isolated |
| 6bf52f30-f3d7-4559-94e4-c59149b5bb42 | 29 | 0 | 12 | 1 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application-isolated |
| 90468c9f-8ee1-4090-b4f4-720c6a8f7592 | 30 | 0 | 16 | 1 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application-isolated |
| 2ff525c9-3dc9-47d7-936b-32177e92c574 | 31 | 0 | 17 | 1 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application-isolated |
| ce4bfea3-54a9-4811-b141-af027c9b19e8 | 32 | 0 | 18 | 1 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application |
| 5c0d13c2-a831-4996-aa6c-8d1ef0945e07 | 33 | 0 | 19 | 1 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application |
| 87095404-512e-46d7-bc91-130fe09df488 | 34 | 0 | 20 | 1 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application |
| be05cb50-6efa-4ab0-a8db-fbfdc5fd6790 | 35 | 0 | 24 | 1 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application |
| 60a8e250-191a-4f08-a2c0-14b948758dd5 | 36 | 0 | 25 | 1 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application |
| 43b8cdce-02e6-41b9-b97c-a79e124439bb | 37 | 0 | 26 | 1 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application |
| 9f5837c0-16c7-46fb-a72d-f07b83a2f7f9 | 38 | 0 | 27 | 1 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application |
| f5a67148-1a36-4d96-b791-7c61d83ad165 | 39 | 0 | 28 | 1 | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz | Application |
+--------------------------------------+-------+-----------+-------+--------+------------------------------------------+----------------------+

[sysadmin@controller-0 ~(keystone_admin)]$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-4.18.0-147.3.1.rt24.96.el8_1.tis.8.x86_64 root=UUID=4b93c74d-1c5a-43b5-8d9f-439e4aa5f83d ro security_profile=standard module_blacklist=integrity,ima audit=0 tboot=false crashkernel=auto biosdevname=0 console=ttyS0,115200 iommu=pt usbcore.autosuspend=-1 selinux=0 enforcing=0 nmi_watchdog=0 softlockup_panic=0 intel_iommu=on user_namespace.enable=1 skew_tick=1 hugepagesz=1G hugepages=40 nopti nospectre_v2 nospectre_v1 hugepagesz=2M hugepages=0 default_hugepagesz=1G irqaffinity=0-1,20-21 rcu_nocbs=2-19,22-39 nohz_full=2-19,22-39 isolcpus=2-11,22-31 kthread_cpus=0-1,20-21

Expected Behavior
------------------

The application-isolated type CPUs can be used by the Kubernetes Pods.

Actual Behavior
----------------
The application-isolated type CPUs can't be used by the Kubernetes Pods and only application type CPUs can be used by Pods.

Reproducibility
---------------
Reproducible

System Configuration
--------------------
One node system

Branch/Pull Time/Commit
-----------------------

[sysadmin@controller-0 ~(keystone_admin)]$ system show
+----------------------+--------------------------------------+
| Property | Value |
+----------------------+--------------------------------------+
| contact | None |
| created_at | 2020-08-10T13:38:52.565086+00:00 |
| description | None |
| https_enabled | False |
| location | None |
| name | ba2ae6d9-a72b-43fc-a181-38eccd27766e |
| region_name | RegionOne |
| sdn_enabled | False |
| security_feature | spectre_meltdown_v1 |
| service_project_name | services |
| software_version | 20.06 |
| system_mode | simplex |
| system_type | All-in-one |
| timezone | UTC |
| updated_at | 2020-08-10T15:46:14.983131+00:00 |
| uuid | 6780937d-804a-4025-a7f6-ee73e57f7436 |
| vswitch_type | none |
+----------------------+--------------------------------------+

Last Pass
---------
No pass

Timestamp/Logs
--------------
N/A

Test Activity
-------------
Sanity

[changyi (chaye)]

Is there any update for this issue? Thanks a lot.

[Sriram Dharwadkar (sriram.ec)]

Is this issue resolved or is there any work around available ? Even we are in need of isolated cpus for kubernetes pods.
Kindly let me know.

[changyi (chaye)]

No, it seems that isolated cpus are not supported by opensource version starlingX.

[Chris Friesen (cbf123)]

Upstream Kubernetes doesn't really know how to handle "isolated" CPUs separately from "normal" CPUs. Because of this sysinv will reserve the "application-isolated" CPUs from use by Kubernetes when using the "static" CPU Manager policy.

If you set the kube-cpu-mgr-policy host label to "none" (via Horizon or the "sysinv" CLI) Kubernetes will use the default policy and should include the "application-isolated" CPUs as part of the "Allocatable" cpu count. You'll have to handle CPU affinity of containerized processes explicitly though to ensure that they don't interfere with each other.

[Ghada Khalil (gkhalil)]

based on notes above from the containers team, this is not fully supported.