incrementing ephemeral-storage requires a lock/unlock

[John Kung]

Brief Description

Incrementing ephemeral-storage requires a lock/unlock or sudo systemctl restart kubelet.

Severity

Minor: System/Feature is usable with minor issue

Steps to Reproduce

system host-fs-modify controller-0 kubelet=20

compare the output for:

system host-fs-list controller-0

with

kubectl describe nodes controller-0 | grep eph

Expected Behavior

kubectl describe nodes controller-0 should show correct amount of memory

Actual Behavior

kubelet has the wrong perception of available memory

Reproducibility

100%

System Configuration

AIOSX should not matter on system type

Branch/Pull Time/Commit

N/A

Last Pass

N/A

Timestamp/Logs

Alarms

Test Activity

Usage

Workaround

1. Document

2. force lock of system when doing system host-fs-modify

3. automatically do sudo systemctl restart when changing kubelet size

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/814138

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/stx-puppet/+/814138
Committed: https://opendev.org/starlingx/stx-puppet/commit/78bb4be97617338a0a2cb278319890e7ac338810
Submitter: "Zuul (22348)"
Branch: master

commit 78bb4be97617338a0a2cb278319890e7ac338810
Author: John Kung <email address hidden>
Date: Fri Oct 15 07:53:54 2021 -0500

    Update kubelet host filesystem ephemeral storage at runtime

    At runtime, restart the kubelet after the ephemeral storage update,
    to allow it to handle the updated ephemeral-storage.

    Test Plan:
    PASSED Verify controller host-fs-modify kubelet
    PASSED Verify worker host-fs-modify kubelet
    PASSED Verify kubectl described node reflects updated ephemeral-storage
    PASSED Verify application pods continue running
    PASSED Verify no stuck Configuration 250.001 alarm or puppet Error

    Closes-Bug: 1947006
    Signed-off-by: John Kung <email address hidden>
    Change-Id: I03ed193678c811274e0a1b595b8ff9450f290cee