fsmond logging the same warning every 3 seconds

[Erickson Silva de Oliveira]

Brief Description
-----------------
mond logging the same warning every 3 seconds. The issue(below warning failed logs) is only present on subclouds, but on all of them (regardless of hardware type).

Reproducibility
---------------
Reproducible

Timestamp/Logs
--------------
sysadmin@controller-0:~$ tail -n 15 /var/log/fsmond.log
2023-11-10T14:59:54.904 [87118.56618] controller-0 fsmond fsm fsmonHdlr.cpp ( 196) fsmon_service : Warn : File (/.fs-test) test failed
2023-11-10T15:00:09.905 [87118.56619] controller-0 fsmond fsm fsmonHdlr.cpp ( 196) fsmon_service : Warn : File (/.fs-test) test failed
2023-11-10T15:00:24.905 [87118.56620] controller-0 fsmond fsm fsmonHdlr.cpp ( 196) fsmon_service : Warn : File (/.fs-test) test failed
2023-11-10T15:00:39.905 [87118.56621] controller-0 fsmond fsm fsmonHdlr.cpp ( 196) fsmon_service : Warn : File (/.fs-test) test failed
2023-11-10T15:00:54.905 [87118.56622] controller-0 fsmond fsm fsmonHdlr.cpp ( 196) fsmon_service : Warn : File (/.fs-test) test failed
2023-11-10T15:01:09.905 [87118.56623] controller-0 fsmond fsm fsmonHdlr.cpp ( 196) fsmon_service : Warn : File (/.fs-test) test failed
2023-11-10T15:01:24.905 [87118.56624] controller-0 fsmond fsm fsmonHdlr.cpp ( 196) fsmon_service : Warn : File (/.fs-test) test failed
2023-11-10T15:01:39.906 [87118.56625] controller-0 fsmond fsm fsmonHdlr.cpp ( 196) fsmon_service : Warn : File (/.fs-test) test failed
2023-11-10T15:01:54.906 [87118.56626] controller-0 fsmond fsm fsmonHdlr.cpp ( 196) fsmon_service : Warn : File (/.fs-test) test failed
2023-11-10T15:02:09.906 [87118.56627] controller-0 fsmond fsm fsmonHdlr.cpp ( 196) fsmon_service : Warn : File (/.fs-test) test failed
2023-11-10T15:02:24.906 [87118.56628] controller-0 fsmond fsm fsmonHdlr.cpp ( 196) fsmon_service : Warn : File (/.fs-test) test failed
2023-11-10T15:02:39.907 [87118.56629] controller-0 fsmond fsm fsmonHdlr.cpp ( 196) fsmon_service : Warn : File (/.fs-test) test failed
2023-11-10T15:02:54.907 [87118.56630] controller-0 fsmond fsm fsmonHdlr.cpp ( 196) fsmon_service : Warn : File (/.fs-test) test failed
2023-11-10T15:03:09.908 [87118.56631] controller-0 fsmond fsm fsmonHdlr.cpp ( 196) fsmon_service : Warn : File (/.fs-test) test failed
2023-11-10T15:03:24.908 [87118.56632] controller-0 fsmond fsm fsmonHdlr.cpp ( 196) fsmon_service : Warn : File (/.fs-test) test failed

--------------------------------------

Enabling debug in /etc/mtc/fsmond.conf gives a little more detail:
```
[debug] ; SIGHUP to reload
debug_timer = 1 ; enable(1) or disable(0) timer logs (tlog)
debug_msg = 1 ; enable(1) or disable(0) message logs (mlog)
debug_state = 1 ; enable(1) or disable(0) state change logs (clog)
debug_level = 15 ; decimal mask 0..15 (8,4,2,1)
debug_all = 1 ;
```
-->
root@controller-0:~# tail -f /var/log/fsmond.log
2023-11-10T14:51:28.543 [3930897.00006] localhost fsmond --- daemon_debug.cpp ( 241) debug_config_handler : Info : State Debug : 1
2023-11-10T14:51:28.543 [3930897.00007] localhost fsmond --- daemon_debug.cpp ( 249) debug_config_handler : Info : Level Debug : f
2023-11-10T14:51:28.543 [3930897.00008] localhost fsmond --- daemon_debug.cpp ( 257) debug_config_handler : Info : Globl Debug : 1
2023-11-10T14:51:28.543 [3930897.00009] localhost fsmond fsm fsmonInit.cpp ( 84) daemon_configure : Info : Interface : vlan637
2023-11-10T14:51:28.543 [3930897.00010] localhost fsmond fsm fsmonInit.cpp ( 86) daemon_configure : Info : Audit Period: 15
2023-11-10T14:51:28.543 [3930897.00011] localhost fsmond com nodeUtil.cpp ( 704) get_hostname : Info : controller-0
2023-11-10T14:51:28.543 [3930897.00012] controller-0 fsmond --- daemon_main.cpp ( 434) main : Info : Build Date : Mon Dec 19 10:59:44 UTC 2022
2023-11-10T14:51:28.543 [3930897.00013] controller-0 fsmond --- daemon_main.cpp ( 435) main : Info : ------------------------------------------------------
2023-11-10T14:51:28.543 [3930897.00014] controller-0 fsmond fsm fsmonHdlr.cpp ( 174) fsmon_service : Info : Starting 'Audit' timer (15 secs)
2023-11-10T14:51:28.543 [3930897.00015] controller-0 fsmond --- nodeTimers.cpp ( 130) _timer_start : Timer: () Tid:0x55fdb2611aa0 with 15.000 second timeout (count:1)
2023-11-10T14:51:43.543 [3930897.00016] controller-0 fsmond fsm fsmonHdlr.cpp ( 96) do_file_test :Debug : Failed to open /.fs-test for writing, error=Operation not permitted.
2023-11-10T14:51:43.543 [3930897.00017] controller-0 fsmond fsm fsmonHdlr.cpp ( 196) fsmon_service : Warn : File (/.fs-test) test failed
2023-11-10T14:51:43.543 [3930897.00018] controller-0 fsmond fsm fsmonHdlr.cpp ( 192) fsmon_service :Debug : File (/scratch/.fs_test) test passed
2023-11-10T14:51:43.543 [3930897.00019] controller-0 fsmond fsm fsmonHdlr.cpp ( 192) fsmon_service :Debug : File (/var/log/.fs_test) test passed
2023-11-10T14:51:43.543 [3930897.00020] controller-0 fsmond fsm fsmonHdlr.cpp ( 192) fsmon_service :Debug : File (/var/run/.fs_test) test passed
2023-11-10T14:51:43.543 [3930897.00021] controller-0 fsmond fsm fsmonHdlr.cpp ( 192) fsmon_service :Debug : File (/var/lock/.fs_test) test passed
2023-11-10T14:51:43.543 [3930897.00022] controller-0 fsmond --- nodeTimers.cpp ( 130) _timer_start : Timer: () Tid:0x55fdb2611aa0 with 15.000 second timeout (count:1)

----------------------------

root@controller-0:~# echo " " >/.valami
-bash: /.valami: Operation not permitted
root@controller-0:~#

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/metal/+/901177

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/metal/+/901177
Committed: https://opendev.org/starlingx/metal/commit/16181a2ce8983bdce433a970b02ebe9416cc065a
Submitter: "Zuul (22348)"
Branch: master

commit 16181a2ce8983bdce433a970b02ebe9416cc065a
Author: Erickson Silva de Oliveira <email address hidden>
Date: Thu Nov 16 12:17:03 2023 -0300

    Replace a file test from fsmond

    fsmond tries to create a test file in "/.fs-test" but
    it is not possible because "/" is blocked by ostree.

    So the fix is to replace this path from fsmond monitoring
    with /sysroot/.fs_test.

    Below is a comparison of the logs:
      - Before change:
      ( 196) fsmon_service : Warn : File (/.fs-test) test failed

      - After change:
      ( 201) fsmon_service : Info : tests passed

    Test Plan:
      - PASS: Build mtce package
      - PASS: Replace fsmond binary on AIO-SX
      - PASS: Check fsmond.log output

    Closes-Bug: 2043712

    Change-Id: Ib4bad73448735bce1dff598151fce86f867f4db7
    Signed-off-by: Erickson Silva de Oliveira <email address hidden>