Comment 1 for bug 1884704

Comments from Bart Wensley:
I don’t know anything about the mgr-restful-plugin, but it seems to be failing to connect to the ceph-mgr process on port 7999 when the issue is happening (from mgr-restful-plugin.log):

2020-06-22 20:12:27,194 3728723 INFO mgr-restful-plugin Run command: /usr/bin/ceph fsid
2020-06-22 20:12:27,446 3728723 INFO mgr-restful-plugin Run command: /usr/bin/ceph auth get mgr.controller-0 -o /var/lib/ceph/mgr/ceph-controller-0/keyring
2020-06-22 20:12:27,714 3728723 INFO mgr-restful-plugin Run command: /usr/bin/ceph config-key get mgr/restful/server_port
2020-06-22 20:12:28,002 3728723 INFO mgr-restful-plugin Run command: /usr/bin/ceph config-key get config/mgr/restful/controller-0/crt
2020-06-22 20:12:28,274 3728723 INFO mgr-restful-plugin Run command: /usr/bin/ceph config-key get mgr/restful/controller-0/crt
2020-06-22 20:12:28,563 3728723 INFO mgr-restful-plugin Run command: /usr/bin/ceph config-key get config/mgr/restful/controller-0/key
2020-06-22 20:12:28,842 3728723 INFO mgr-restful-plugin Run command: /usr/bin/ceph config-key get /mgr/restful/controller-0/key
2020-06-22 20:12:29,118 3728723 INFO mgr-restful-plugin Create restful plugin self signed certificate
2020-06-22 20:12:29,279 3728723 INFO mgr-restful-plugin Run command: /usr/bin/ceph config-key set config/mgr/restful/controller-0/crt -i /tmp/tmp4XO7c4/crt
2020-06-22 20:12:29,561 3728723 INFO mgr-restful-plugin Run command: /usr/bin/ceph config-key set mgr/restful/controller-0/crt -i /tmp/tmp4XO7c4/crt
2020-06-22 20:12:29,845 3728723 INFO mgr-restful-plugin Run command: /usr/bin/ceph config-key set config/mgr/restful/controller-0/key -i /tmp/tmp4XO7c4/key
2020-06-22 20:12:30,123 3728723 INFO mgr-restful-plugin Run command: /usr/bin/ceph config-key set mgr/restful/controller-0/key -i /tmp/tmp4XO7c4/key
2020-06-22 20:12:30,393 3728723 INFO mgr-restful-plugin Stop unmanaged running ceph-mgr processes
2020-06-22 20:12:30,491 3728723 INFO mgr-restful-plugin Start ceph-mgr daemon
2020-06-22 20:12:45,509 3728723 INFO mgr-restful-plugin Run command: /usr/bin/ceph mgr module ls --format json
2020-06-22 20:12:48,781 3728723 INFO mgr-restful-plugin Run command: /usr/bin/ceph config-key get mgr/restful/keys/admin
2020-06-22 20:12:49,057 3728723 INFO mgr-restful-plugin Run command: /usr/bin/ceph mgr services --format json
2020-06-22 20:12:49,320 3728723 INFO mgr-restful-plugin Run command: /usr/bin/ceph config-key get config/mgr/restful/controller-0/crt
2020-06-22 20:13:28,726 3728723 WARNING mgr-restful-plugin REST API ping failed: reason=HTTPSConnectionPool(host='controller-0', port=7999): Read timed out. (read timeout=15)
2020-06-22 20:13:28,726 3728723 INFO mgr-restful-plugin REST API ping failure count=0
2020-06-22 20:13:28,726 3728723 INFO mgr-restful-plugin Run command: /usr/bin/ceph fsid
2020-06-22 20:13:47,003 3728723 WARNING mgr-restful-plugin REST API ping failed: reason=HTTPSConnectionPool(host='controller-0', port=7999): Read timed out. (read timeout=15)
2020-06-22 20:13:47,004 3728723 INFO mgr-restful-plugin REST API ping failure count=1
2020-06-22 20:13:47,004 3728723 INFO mgr-restful-plugin Run command: /usr/bin/ceph fsid
2020-06-22 20:15:22,719 3728723 WARNING mgr-restful-plugin REST API ping failed: reason=HTTPSConnectionPool(host='controller-0', port=7999): Read timed out. (read timeout=15)
2020-06-22 20:15:22,719 3728723 INFO mgr-restful-plugin REST API ping failure count=0
2020-06-22 20:15:22,719 3728723 INFO mgr-restful-plugin Run command: /usr/bin/ceph fsid
2020-06-22 20:15:41,008 3728723 WARNING mgr-restful-plugin REST API ping failed: reason=HTTPSConnectionPool(host='controller-0', port=7999): Read timed out. (read timeout=15)
2020-06-22 20:15:41,009 3728723 INFO mgr-restful-plugin REST API ping failure count=1
2020-06-22 20:15:41,009 3728723 INFO mgr-restful-plugin Run command: /usr/bin/ceph fsid
2020-06-22 20:16:28,480 3728723 WARNING mgr-restful-plugin REST API ping failed: reason=HTTPSConnectionPool(host='controller-0', port=7999): Read timed out. (read timeout=15)
2020-06-22 20:16:28,481 3728723 INFO mgr-restful-plugin REST API ping failure count=0
2020-06-22 20:16:42,306 3728709 INFO mgr-restful-plugin Restful plugin does not respond but failure count is within acceptable limits: ceph_mgr=0 < 3, ping=0 < 5. Report status OK
2020-06-22 20:16:42,307 3728709 WARNING mgr-restful-plugin Failed to send response back. request=status, response=OK, reason=[Errno 32] Broken pipe
2020-06-22 20:16:42,307 3728723 INFO mgr-restful-plugin Run command: /usr/bin/ceph fsid
2020-06-22 20:16:57,329 3728709 INFO mgr-restful-plugin Restful plugin does not respond but failure count is within acceptable limits: ceph_mgr=0 < 3, ping=1 < 5. Report status OK
2020-06-22 20:16:57,329 3728709 WARNING mgr-restful-plugin Failed to send response back. request=status, response=OK, reason=[Errno 32] Broken pipe
2020-06-22 20:16:57,329 3728709 INFO mgr-restful-plugin Stop monitor with SIGTERM to process group 3728723
2020-06-22 20:17:00,606 3728723 WARNING mgr-restful-plugin REST API ping failed: reason=HTTPSConnectionPool(host='controller-0', port=7999): Read timed out. (read timeout=15)
2020-06-22 20:17:00,607 3728723 INFO mgr-restful-plugin REST API ping failure count=1
2020-06-22 20:17:02,335 3728709 INFO mgr-restful-plugin Stop monitor with SIGKILL to process group 3728723
2020-06-22 20:17:02,336 3728709 INFO mgr-restful-plugin Monitor stopped: pid=3728723
2020-06-22 20:17:02,337 3728709 INFO mgr-restful-plugin Remove service pid file: path=/var/run/ceph/mgr-restful-plugin.pid
2020-06-22 20:17:02,337 3728709 INFO mgr-restful-plugin Close service socket and remove file: path=/var/run/ceph/mgr/mgr-restful-plugin.socket
2020-06-22 20:17:02,337 3728709 INFO mgr-restful-plugin Release service lock: path=/var/run/ceph/mgr/mgr-restful-plugin.lock