| 2023-02-02 21:27:15 |
Mark Cunningham |
description |
A client has reported that they had a few machines with an excessive number of message files in the /var/lib/landscape/messages/ directory, which consumed all the inodes in the machines, and caused a full disk condition. This seems to have been the result of the client machines having an issue contacting the Landscape server for an indeterminate/extended amount of time.
The request here is to add a disk capacity sanity check into the landscape-client to prevent this kind of thing in the future. Perhaps both a check of disk space used as well as an inode usage check. If this gets over some threshold (yet to be determined), the Landscape client should change behavior accordingly.
One potential suggestion could be to stop creating new messages or rotate/compress the old messages into a bundle for later transport. Conversely perhaps there could be just a check on the count of pending messages, and if this gets over some threshold (also yet to be determined) new messages could be stopped, pending messages could be purged and a resync action initiated. New messages could be started normally again after the resync action completes. |
A client has reported that they had a few machines with an excessive number of message files in the /var/lib/landscape/client/messages/ directory, which consumed all the inodes in the machines, and caused a full disk condition. This seems to have been the result of the client machines having an issue contacting the Landscape server for an indeterminate/extended amount of time.
The request here is to add a disk capacity sanity check into the landscape-client to prevent this kind of thing in the future. Perhaps both a check of disk space used as well as an inode usage check. If this gets over some threshold (yet to be determined), the Landscape client should change behavior accordingly.
One potential suggestion could be to stop creating new messages or rotate/compress the old messages into a bundle for later transport. Conversely perhaps there could be just a check on the count of pending messages, and if this gets over some threshold (also yet to be determined) new messages could be stopped, pending messages could be purged and a resync action initiated. New messages could be started normally again after the resync action completes. |
|
