stx.2.0: StarlingX Duplex AIO not utilizing its full quota available

[ANIRUDH GUPTA]

Brief Description
-----------------------------
I have deployed StarlingX 2.0 AIO Duplex Bare metal setup and trying to launch VM's on it.
Each Server Configuration:
40 cores
64 GB RAM

Scenario 1: Each VM with 6 cores

As per the container "quay.io/airshipit/armada:dd2e56c473549fd16f94212b553ed58c48d1f51b-ubuntu_bionic", the detected cores are 40 only.

{"log":"detected number of CPU cores: 40\n","stream":"stderr","time":"2020-01-28T07:23:04.872496423Z"}

But at the time of vm creation, i am seeing the below error in container "k8s_nova-osapi_nova-api-osapi-7664b779-lxdjf_openstack_51abe779-424d-11ea-ad09-f8f21e64b290_0" and "k8s_nova-api-proxy_nova-api-proxy-77477478df-bhlkf_openstack_4fe71bcd-424d-11ea-ad09-f8f21e64b290_0"

20-01-29 09:03:41.546 1 INFO nova.api.openstack.wsgi [req-0d53be9c-f6a6-4010-80a2-2fe5a42352bb 9d2ac3babe1f4a1393425332ed8cd935 f4aae7795f41446db259013319b2b393 - default default] HTTP exception thrown: Quota exceeded for cores: Requested 6, but already used 18 of 20 cores\n","stream":"stdout","time":"2020-01-29T09:03:41.547384563Z"}

Scenario 2: Each VM with 7GB RAM

As per the hypervisor overview, I have 128 GB of total RAM available (~62 GB on each server)
But when I try to spawn multiple VM's with each 7GB RAM, then I am getting the error

{"log":"2020-01-29 11:19:45,570.570 6 INFO nova_api_proxy.apps.proxy [-] POST response body: ['{\"forbidden\": {\"message\": \"Quota exceeded for ram: Requested 7168, but already used 50176 of 51200 ram\", \"code\": 403}}']\n","stream":"stdout","time":"2020-01-29T11:19:45.571928857Z"}
{"log":"2020-01-29 11:19:52,114.114 6 INFO nova_api_proxy.apps.acceptor [-] POST request issued by user (admin) tenant (admin) remote address (172.16.192.64) \"POST http://nova.openstack.svc.cluster.local/v2.1/f4aae7795f41446db259013319b2b393/servers\"\n","stream":"stdout","time":"2020-01-29T11:19:52.114827364Z"}

The logs and required screenshot is attached in the mail

[Ghada Khalil (gkhalil)]

Assigning to distro.openstack PL for review. Appears to be a quota or possibly over-subscription issue (perhaps NUMA related) related to nova

[zhipeng liu (zhipengs)]

Hi Anirudh,

The log you provided is not the key log for trouble shooting.
Please reproduce it and provide all nova related log in /var/log/containers/

BTW, before creating new VMs, please use below command to make sure no other error VM existed.
openstack server list

Thanks!
Zhipeng

[Zhang Kunpeng (zhangkunpeng)]

I think the core quota of the project may be only 20.
Please run the follow commond to check the project quota

openstack quota show <your project name> --class

or you can find the quota in 'identify > Project' by browser

[ANIRUDH GUPTA (anyrude10)]

Hi,

Please find attached the logs of controller-0 and the output required.

Due to size constraint, logs of controller-1 will be attached in the next rep.

openstack quota show admin --class
+----------------------+-------+
| Field | Value |
+----------------------+-------+
| backup-gigabytes | 1000 |
| backups | 10 |
| cores | -1 |
| fixed-ips | -1 |
| floating-ips | 10 |
| gigabytes | 1000 |
| gigabytes_ceph-store | -1 |
| groups | 10 |
| injected-file-size | 10240 |
| injected-files | 5 |
| injected-path-size | 255 |
| instances | 100 |
| key-pairs | 100 |
| per-volume-gigabytes | -1 |
| project | admin |
| project_name | admin |
| properties | 128 |
| ram | -1 |
| secgroup-rules | 20 |
| secgroups | 10 |
| snapshots | 10 |
| snapshots_ceph-store | -1 |
| volumes | 10 |
| volumes_ceph-store | -1 |
+----------------------+-------+

[ANIRUDH GUPTA (anyrude10)]

As described in the above reply, Please find the logs of controller-1

[Zhang Kunpeng (zhangkunpeng)]

Could you update the cores quota to 40 and try to launch the VMs again.

[ANIRUDH GUPTA (anyrude10)]

Hi,

This is a StarlingX duplex baremetal 2.0 with each server having 40 cores.

Even after setting the value to be 40, same error is being faced.

Attaching controller-0 logs here and controller-1 in the subsequent reply.

controller-0:~$ openstack quota show admin --class
+----------------------+-------+
| Field | Value |
+----------------------+-------+
| backup-gigabytes | 1000 |
| backups | 10 |
| cores | 40 |
| fixed-ips | -1 |
| floating-ips | 10 |
| gigabytes | 1000 |
| gigabytes_ceph-store | -1 |
| groups | 10 |
| injected-file-size | 10240 |
| injected-files | 5 |
| injected-path-size | 255 |
| instances | 100 |
| key-pairs | 100 |
| per-volume-gigabytes | -1 |
| project | admin |
| project_name | admin |
| properties | 128 |
| ram | -1 |
| secgroup-rules | 20 |
| secgroups | 10 |
| snapshots | 10 |
| snapshots_ceph-store | -1 |
| volumes | 10 |
| volumes_ceph-store | -1 |
+----------------------+-------+

[ANIRUDH GUPTA (anyrude10)]

As described in the above reply, Please find the logs of controller-1

[ANIRUDH GUPTA (anyrude10)]

Hi Team,

With the shared logs, is there any update on the issue?

[zhipeng liu (zhipengs)]

Hi Kunpeng,

Do you have any update on the latest log, thanks!

Zhipeng

[Zhang Kunpeng (zhangkunpeng)]

I'm sure this issue is caused by the admin project quota. vcpu default quota is 20 and memery default quota is 51200. If the vcpu count of lanched VMs is more than 20, it raises the same error and stop to launch the lastest VM, so does memery.