Logs are hammered with a healtheck error in empty /var/log/containers/stdouts/healthcheck.log

Bug #1902602 reported by Pete Zaitcev
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
New
Undecided
Unassigned

Bug Description

On undercloud node, /var/log/messages is overflowing with messages like these:

Nov 2 00:00:13 undercloud podman[818647]: 2020-11-02 00:00:13.013180118 -0500 EST m=+0.892513654 container exec 4218e223289e3dd84e044e0c7c647eed680b1c8c5911be12f79c0f292343e77f (image=undercloud.ctlplane.tripleo1.zaitcev.usersys.redhat.com:8787/tripleomaster/openstack-heat-api:current-tripleo, name=heat_api_cron)
Nov 2 00:00:13 undercloud podman[818629]: healthy
Nov 2 00:00:13 undercloud podman[818629]: Error: unable to update health check log /var/log/containers/stdouts/healthcheck.log for 2849cfe4117b4eb52cee03ea56fa5c161efcfa9ca1f276e8785bdec935aa0493: failed to unmarshal existing healthcheck results in /var/log/containers/stdouts/healthcheck.log: readObjectStart: expect { or n, but found
Nov 2 00:00:13 undercloud podman[818629]: , error found in #0 byte of ...||..., bigger context ...||...
Nov 2 00:00:13 undercloud systemd[1]: 2849cfe4117b4eb52cee03ea56fa5c161efcfa9ca1f276e8785bdec935aa0493.service: Main process exited, code=exited, status=125/n/a
Nov 2 00:00:13 undercloud systemd[1]: 2849cfe4117b4eb52cee03ea56fa5c161efcfa9ca1f276e8785bdec935aa0493.service: Failed with result 'exit-code'.
Nov 2 00:00:13 undercloud podman[818647]: healthy
Nov 2 00:00:13 undercloud podman[818647]: Error: unable to update health check log /var/log/containers/stdouts/healthcheck.log for 4218e223289e3dd84e044e0c7c647eed680b1c8c5911be12f79c0f292343e77f: failed to unmarshal existing healthcheck results in /var/log/containers/stdouts/healthcheck.log: readObjectStart: expect { or n, but found
Nov 2 00:00:13 undercloud podman[818647]: , error found in #0 byte of ...||..., bigger context ...||...

The rate of these messages is several such blocks EACH SECOND.

They can be suppressed for a short period of time by doing
echo '{}' > /var/log/containers/stdouts/healthcheck.log
but they soon begin anew as something truncates the file.

This is happening with the trunk tripleo, installed with
"tripleo-repos current" (with no -b of any kind).

I am unable even to figure out which component is responsible for parsing the JSON
in the log, and/or which component truncates the file repeatedly.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.