Hi,
With u-r-c deployed in GCE, units have sshguard installed. This can cause issues with the ssh-keyscan used to build a list of ssh known hosts as well as the metadata sync, which uses rsync over ssh. We should check and purge this on install.
Failure with ssh-keyscan:
| 2021-03-01 02:29:25 INFO juju-log cluster:2: Syncing authorized_keys @ /home/www-sync/.ssh/authorized_keys.
| 2021-03-01 02:29:35 WARNING cluster-relation-changed # 10.240.0.9:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.10
| 2021-03-01 02:29:35 WARNING cluster-relation-changed # 10.240.0.9:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.10
| 2021-03-01 02:29:35 WARNING cluster-relation-changed # 10.240.0.9:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.10
| 2021-03-01 02:29:50 INFO juju-log cluster:2: Syncing known_hosts @ /home/www-sync/.ssh/known_hosts.
vs.
| 2021-03-01 02:55:36 INFO juju-log cluster:2: Syncing authorized_keys @ /home/www-sync/.ssh/authorized_keys.
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.40:22 SSH-2.0-OpenSSH_7.1p2 Ubuntu-2.IS.PATCHED.16.04.4
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.40:22 SSH-2.0-OpenSSH_7.1p2 Ubuntu-2.IS.PATCHED.16.04.4
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.40:22 SSH-2.0-OpenSSH_7.1p2 Ubuntu-2.IS.PATCHED.16.04.4
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.41:22 SSH-2.0-OpenSSH_7.1p2 Ubuntu-2.IS.PATCHED.16.04.4
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.41:22 SSH-2.0-OpenSSH_7.1p2 Ubuntu-2.IS.PATCHED.16.04.4
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.41:22 SSH-2.0-OpenSSH_7.1p2 Ubuntu-2.IS.PATCHED.16.04.4
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.9:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.10
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.9:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.10
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.9:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.10
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.42:22 SSH-2.0-OpenSSH_7.1p2 Ubuntu-2.IS.PATCHED.16.04.4
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.42:22 SSH-2.0-OpenSSH_7.1p2 Ubuntu-2.IS.PATCHED.16.04.4
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.42:22 SSH-2.0-OpenSSH_7.1p2 Ubuntu-2.IS.PATCHED.16.04.4
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.2:22 SSH-2.0-OpenSSH_7.1p2 Ubuntu-2.IS.PATCHED.16.04.4
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.2:22 SSH-2.0-OpenSSH_7.1p2 Ubuntu-2.IS.PATCHED.16.04.4
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.2:22 SSH-2.0-OpenSSH_7.1p2 Ubuntu-2.IS.PATCHED.16.04.4
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.6:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.10
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.6:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.10
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.6:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.10
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.8:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.10
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.8:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.10
| 2021-03-01 02:55:36 WARNING cluster-relation-changed # 10.240.0.8:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.10
| 2021-03-01 02:55:36 INFO juju-log cluster:2: Syncing known_hosts @ /home/www-sync/.ssh/known_hosts.
Failure with metadata sync:
| 2021-03-01 03:35:16 INFO juju-log cluster:2: ubuntu_active currently points at: /srv/ubuntu-repository-cache/apache/data/ubuntu_2021-03-01_02:25:01_u10
| 2021-03-01 03:35:16 INFO juju-log cluster:2: Keeping /srv/ubuntu-repository-cache/apache/data/ubuntu_2021-03-01_02:25:01_u10
| 2021-03-01 03:37:26 WARNING cluster-relation-changed ssh: connect to host 10.240.0.9 port 22: Connection timed out
| 2021-03-01 03:37:26 ERROR juju-log cluster:2: Hook error:
| Traceback (most recent call last):
| File "/var/lib/juju/agents/unit-ubuntu-repository-cache-11/.venv/lib/python3.5/site-packages/charms/reactive/__init__.py", line 74, in main
| bus.dispatch(restricted=restricted_mode)
| File "/var/lib/juju/agents/unit-ubuntu-repository-cache-11/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 379, in dispatch
| _invoke(hook_handlers)
| File "/var/lib/juju/agents/unit-ubuntu-repository-cache-11/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 359, in _invoke
| handler.invoke()
| File "/var/lib/juju/agents/unit-ubuntu-repository-cache-11/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 181, in invoke
| self._action(*args)
| File "/var/lib/juju/agents/unit-ubuntu-repository-cache-11/charm/reactive/ubuntu_repository_cache.py", line 211, in cluster_relation_changed
| mirror.peer_update_metadata()
| File "/var/lib/juju/agents/unit-ubuntu-repository-cache-11/charm/lib/ubuntu_repository_cache/mirror.py", line 296, in peer_update_metadata
| _nonleader_update_metadata()
| File "/var/lib/juju/agents/unit-ubuntu-repository-cache-11/charm/lib/ubuntu_repository_cache/mirror.py", line 251, in _nonleader_update_metadata
| if util.remote_path_exists('www-sync', leader_ip, canary):
| File "lib/ubuntu_repository_cache/util.py", line 238, in remote_path_exists
| subprocess.check_call(cmd)
| File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
| raise CalledProcessError(retcode, cmd)
| subprocess.CalledProcessError: Command '('sudo', '-u', 'www-sync', 'ssh', 'www-sync@10.240.0.9', 'stat /srv/ubuntu-repository-cache/apache/data/ubuntu_2021-03-01_03:25:02_u10_good')' returned non-zero exit status 255
