openstack train - gnocchi_statsd health check fails on controllers "There is no gnocchi-statsd process listening on ports 8125 in the container"

Bug #1907485 reported by James Bagwell
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo-common
In Progress
Undecided
Martin Mágr

Bug Description

Description
===========
After successful deployment of Openstack Train, gnocchi_statsd container running on controllers is in unhealthy state.

Error of health check : There is no gnocchi-statsd process listening on ports 8125 in the container

Steps to reproduce
==================
Deploy fresh Openstack train deployment.
Check container status's on controller.
sudo docker ps | grep gnocchi_statsd

Expected result
===============
I would expect that the gnocchi_statsd container would be healthy after its health check.

Actual result
=============
gnocchi_statsd container is unhealthy.

Environment
===========
1. Openstack Train using /master branch health checks (lsof instead of ss)

2. Ceph

3. ovs

Logs & Configs
==============

[cbis-admin@overcloud-controller-bagwell-cloud-0 ~]$ sudo docker ps | grep gnocchi_statsd
a13bae5a6549 172.31.0.1:5001/cbis/centos-binary-gnocchi-statsd:train-latest "dumb-init --singl..." 5 hours ago Up About an hour (unhealthy) gnocchi_statsd

Running check from outside the container:
[cbis-admin@overcloud-controller-bagwell-cloud-0 ~]$ sudo docker exec -u root gnocchi_statsd /openstack/healthcheck
There is no gnocchi-statsd process listening on ports 8125 in the container.

Running check from inside the container:
https://pastebin.com/G8EXY3iX

Looking at LSOF output: ( please note /proc/8/ PERMISSION DENIED )

https://pastebin.com/nLnUYD9U

Revision history for this message
James Bagwell (jimbagwell) wrote :
affects: tripleo → tripleo-common
Revision history for this message
James Bagwell (jimbagwell) wrote :

@cedric - can you assist here?

summary: - gnocchi_statd health check fails
+ openstack train - gnocchi_statd health check fails on controllers
summary: - openstack train - gnocchi_statd health check fails on controllers
+ openstack train - gnocchi_statsd health check fails on controllers
description: updated
summary: openstack train - gnocchi_statsd health check fails on controllers
+ "There is no gnocchi-statsd process listening on ports 8125 in the
+ container"
Revision history for this message
Cédric Jeanneret (cjeanner) wrote :

Hello,

weird thing - since the healthcheck (and your `docker exec') is supposed to be launched as root, I don't really see why there is any permission denied on the lsof...
I note you're using docker - what's the centos version here?

Cheers,

C.

Changed in tripleo-common:
assignee: nobody → Cédric Jeanneret (cjeanner)
Revision history for this message
Cédric Jeanneret (cjeanner) wrote :

We need some more information about the OS and versions installed on the host.

Changed in tripleo-common:
status: New → Incomplete
Revision history for this message
Cédric Jeanneret (cjeanner) wrote :

Apparently the real issue is "UDP vs TCP":
https://review.opendev.org/c/openstack/tripleo-common/+/767293

Changed in tripleo-common:
assignee: Cédric Jeanneret (cjeanner) → nobody
status: Incomplete → In Progress
Martin Mágr (mmagr)
Changed in tripleo-common:
assignee: nobody → Martin Mágr (mmagr)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.