host-unlock compute node rejected: Total allocated memory exceeds the total memory

[Peng Peng]

Brief Description
-----------------
In regular lowlatency system. host-lock compute success, but host-unlock rejected by Total allocated memory exceeds the total memory of compute-1 numa node 0

Severity
--------
Major

Steps to Reproduce
------------------
as description

TC-name: mtc/test_lock_unlock_host.py::test_lock_unlock_host[compute]

Expected Behavior
------------------

Actual Behavior
----------------

Reproducibility
---------------
Seen once

System Configuration
--------------------
Multi-node system

Lab-name: Ip_1-4

Branch/Pull Time/Commit
-----------------------
stx master as of 20190724T013000Z

Last Pass
---------
Lab: IP_1_4
Load: 20190721T233000Z

Timestamp/Logs
--------------
[2019-07-24 08:48:43,872] 301 DEBUG MainThread ssh.send :: Send 'fm --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne alarm-list --nowrap --uuid'
[2019-07-24 08:48:45,246] 423 DEBUG MainThread ssh.expect :: Output:

[sysadmin@controller-0 ~(keystone_admin)]$

[2019-07-24 08:48:45,685] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list'
[2019-07-24 08:48:47,402] 423 DEBUG MainThread ssh.expect :: Output:
+---------------------+--------------------------------+-------------------------------+--------------------+---------+-----------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+---------+-----------+
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-17-centos-stable-versioned | armada-manifest | stx-openstack.yaml | applied | completed |
+---------------------+--------------------------------+-------------------------------+--------------------+---------+-----------+
[sysadmin@controller-0 ~(keystone_admin)]$

[2019-07-24 08:48:50,973] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-lock compute-1'

[2019-07-24 08:49:33,429] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-show compute-1'
[2019-07-24 08:49:35,049] 423 DEBUG MainThread ssh.expect :: Output:
+---------------------+-----------------------------------------------------------------+
| Property | Value |
+---------------------+-----------------------------------------------------------------+
| action | none |
| administrative | locked |
| availability | online |
| bm_ip | None |
| bm_type | None |
| bm_username | None |
| boot_device | /dev/disk/by-path/pci-0000:04:00.0-sas-0x5001e674e0243000-lun-0 |
| capabilities | {} |
| config_applied | 3b0daed7-01c8-42d2-864b-ccd88580987c |
| config_status | None |
| config_target | 3b0daed7-01c8-42d2-864b-ccd88580987c |
| console | ttyS0,115200n8 |
| created_at | 2019-07-24T06:51:38.660631+00:00 |
| hostname | compute-1 |
| id | 2 |
| install_output | text |
| install_state | completed |
| install_state_info | None |
| invprovision | provisioned |
| location | {} |
| mgmt_ip | 192.168.204.149 |
| mgmt_mac | 00:1e:67:4e:02:44 |
| operational | disabled |
| personality | worker |
| reserved | False |
| rootfs_device | /dev/disk/by-path/pci-0000:04:00.0-sas-0x5001e674e0243000-lun-0 |
| serialid | None |
| software_load | 19.01 |
| subfunctions | worker,lowlatency |
| task | |
| tboot | false |
| ttys_dcd | None |
| updated_at | 2019-07-24T08:49:33.446828+00:00 |
| uptime | 5551 |
| uuid | 4e91f428-66fa-4f6e-97d8-2f46d3df0f49 |
| vim_progress_status | services-disabled |
+---------------------+-----------------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$

[2019-07-24 08:49:37,142] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock compute-1'
[2019-07-24 08:49:39,203] 423 DEBUG MainThread ssh.expect :: Output:
Rejected: Total allocated memory exceeds the total memory of compute-1 numa node 0
[sysadmin@controller-0 ~(keystone_admin)]$

Test Activity
-------------
Sanity

[Tao Liu (tliu88)]

I took a look at IP 1-4 and found that compute-1 numa node 0 total memory has decreased from 15478 MiB to 14584 MiB between the 2 unlock actions. The second one was rejected due to “Total allocated memory exceeds the total memory”. There is no log to display the node free memory retrieved from Linux on the agent side and it is hard to tell why the available memory was reduced.

# first unlock
2019-07-24 07:12:50.136 104744 INFO sysinv.api.controllers.v1.host [-] compute-1 ihost check_unlock_worker
2019-07-24 07:12:50.559 104744 INFO sysinv.api.controllers.v1.host [-] Memory: Total=15478 MiB, Allocated=8000 MiB, 2M: 7739 pages None pages pending, 1G: 15 pages None pages pending
2019-07-24 07:12:50.593 104744 INFO sysinv.api.controllers.v1.host [-] Memory: Total=15745 MiB, Allocated=2000 MiB, 2M: 7872 pages None pages pending, 1G: 15 pages None pages pending
2019-07-24 07:12:50.615 104744 INFO sysinv.api.controllers.v1.host [-] host(compute-1) node(5): vm_mem_mib=6454,vm_mem_mib_possible (from agent) = 15478
2019-07-24 07:12:50.615 104744 INFO sysinv.api.controllers.v1.host [-] Updating mem values of host(compute-1) node(5): {'vm_hugepages_nr_4K': 165376, 'vm_hugepages_nr_2M': 2904, 'vswitch_hugepages_nr': 1}
2019-07-24 07:12:50.740 104744 INFO sysinv.api.controllers.v1.host [-] host(compute-1) node(6): vm_mem_mib=12721,vm_mem_mib_possible (from agent) = 15744
2019-07-24 07:12:50.741 104744 INFO sysinv.api.controllers.v1.host [-] Updating mem values of host(compute-1) node(6): {'vm_hugepages_nr_4K': 325888, 'vm_hugepages_nr_2M': 5724, 'vswitch_hugepages_nr': 1}

# second unlock
2019-07-24 08:49:38.825 104743 INFO sysinv.api.controllers.v1.host [-] compute-1 ihost check_unlock_worker
2019-07-24 08:49:39.153 104743 INFO sysinv.api.controllers.v1.host [-] Memory: Total=14584 MiB, Allocated=14832 MiB, 2M: 2780 pages None pages pending, 1G: 5 pages None pages pending
2019-07-24 08:49:39.154 104743 INFO sysinv.api.controllers.v1.host [-] host unlock check didn't pass, so set the ihost_action back to None and re-raise the exception
2019-07-24 08:49:39.166 104743 WARNING wsme.api [-] Client-side error: Rejected: Total allocated memory exceeds the total memory of compute-1 numa node 0

[Ghada Khalil (gkhalil)]

Waiting to see if this issue is reproducible before deciding on priority/gate

[Ghada Khalil (gkhalil)]

Based on discussion with Brent and Dariush, marking as stx.2.0 medium priority.

The recommendation is to have the software add a safety margin and not allow the user to fully allocate the free memory. This should increase the chance that the subsequent unlock passing.

[Senthil Mukundakumar (smukunda)]

This issue also applies to AIO_DX. Reproduced in daily sanity.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/674718

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/674718
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=e2cc0f53ac7be50fc4dfd1e2baedc3b2b32b1038
Submitter: Zuul
Branch: master

commit e2cc0f53ac7be50fc4dfd1e2baedc3b2b32b1038
Author: Tao Liu <email address hidden>
Date: Thu Aug 1 14:32:57 2019 -0400

    Fix the host unlock rejection caused by memory check

    Historically, the node total memory is calculated based on
    total of allocated hugepages, and usable Linux memory.
    NODE_TOTAL_MiB = HTOT + (AVAIL + PSS)

    This bottom-up calculation introduced variability and complexity.
    In some cases, the previously allocated memory exceeds the current
    total memory when AVAIL and PSS changed.

    This update changes NODE_TOTAL_MiB to MemTotal (the node's total
    usable ram), therefore Possible Huge Pages Memory = MemTotal -
    Platform Reserved.

    This update also adds a constraint, which only allows 90% of the
    Possible Huge Pages Memory to be allocated.

    Closes-Bug: 1837749

    Change-Id: I2fa1d82f70263bb2a1b93df2a405a979b42ed83a
    Signed-off-by: Tao Liu <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.2.0
Review: https://review.opendev.org/675996

[Peng Peng (ppeng)]

The issue was not reproduced recently.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/675996
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=48a3c500ba714bee1d16358af9926667d3f0bb76
Submitter: Zuul
Branch: r/stx.2.0

commit 48a3c500ba714bee1d16358af9926667d3f0bb76
Author: Tao Liu <email address hidden>
Date: Thu Aug 1 14:32:57 2019 -0400

    Fix the host unlock rejection caused by memory check

    Historically, the node total memory is calculated based on
    total of allocated hugepages, and usable Linux memory.
    NODE_TOTAL_MiB = HTOT + (AVAIL + PSS)

    This bottom-up calculation introduced variability and complexity.
    In some cases, the previously allocated memory exceeds the current
    total memory when AVAIL and PSS changed.

    This update changes NODE_TOTAL_MiB to MemTotal (the node's total
    usable ram), therefore Possible Huge Pages Memory = MemTotal -
    Platform Reserved.

    This update also adds a constraint, which only allows 90% of the
    Possible Huge Pages Memory to be allocated.

    Closes-Bug: 1837749

    Change-Id: I2fa1d82f70263bb2a1b93df2a405a979b42ed83a
    Signed-off-by: Tao Liu <email address hidden>