Comment 7 for bug 1754682

Revision history for this message
Ryan Beisner (1chb1n) wrote :

I know you've destroyed and are about to redeploy with latest wares, and the following relates to the initial deploy referenced in this bug. But pasting observations thus-far:

The traceback in the uploaded file is completely different than that shown in the bug description. While it may be related, I'd like to see the unit log and keystone deamon logs for the same timeframe matching the traceback from the bug description (or a repro of that).

Also, I'm not able to determine what the .166 address is from the model status or bundles pastes, but this is a problem:
ssh: connect to host 10.243.8.166 port 22: Connection timed out

This could be a network or MTU issue. If the issue it recurs with port 22 connect timeouts, please confirm switchport MTUs are high enough for NIC MTUs and container interface MTUs.

https://pastebin.canonical.com/p/jGGZXbH2m6/

and

https://pastebin.canonical.com/p/zP9xZZcGps/

...

2018-03-09 15:34:03 INFO juju-log cluster:22: Error syncing remote files
2018-03-09 15:34:03 DEBUG cluster-relation-changed Traceback (most recent call last):
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-changed", line 880, in <module>
2018-03-09 15:34:03 DEBUG cluster-relation-changed main()
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-changed", line 873, in main
2018-03-09 15:34:03 DEBUG cluster-relation-changed hooks.execute(sys.argv)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/hookenv.py", line 800, in execute
2018-03-09 15:34:03 DEBUG cluster-relation-changed self._hooks[hook_name]()
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1449, in wrapped_f
2018-03-09 15:34:03 DEBUG cluster-relation-changed restart_functions)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/host.py", line 730, in restart_on_change_helper
2018-03-09 15:34:03 DEBUG cluster-relation-changed r = lambda_f()
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1448, in <lambda>
2018-03-09 15:34:03 DEBUG cluster-relation-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1650, in _inner_update_certs_if_available
2018-03-09 15:34:03 DEBUG cluster-relation-changed return f(*args, **kwargs)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-changed", line 618, in cluster_changed
2018-03-09 15:34:03 DEBUG cluster-relation-changed update_all_identity_relation_units_force_sync()
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1824, in inner_synchronize_ca_if_changed2
2018-03-09 15:34:03 DEBUG cluster-relation-changed peer_settings = synchronize_ca(fatal=fatal)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1724, in synchronize_ca
2018-03-09 15:34:03 DEBUG cluster-relation-changed synced_units = unison_sync([SSL_SYNC_ARCHIVE, SYNC_FLAGS_DIR])
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
2018-03-09 15:34:03 DEBUG cluster-relation-changed return f(*args, **kwargs)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1445, in unison_sync
2018-03-09 15:34:03 DEBUG cluster-relation-changed fatal=True)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/unison/__init__.py", line 314, in sync_to_peers
2018-03-09 15:34:03 DEBUG cluster-relation-changed sync_to_peer(host, user, paths, verbose, cmd, gid, fatal)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/unison/__init__.py", line 300, in sync_to_peer
2018-03-09 15:34:03 DEBUG cluster-relation-changed sync_path_to_host(p, host, user, verbose, cmd, gid, fatal)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/unison/__init__.py", line 285, in sync_path_to_host
2018-03-09 15:34:03 DEBUG cluster-relation-changed run_as_user(user, cmd, gid)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/unison/__init__.py", line 239, in run_as_user
2018-03-09 15:34:03 DEBUG cluster-relation-changed return check_output(cmd, preexec_fn=_run_as_user(user, gid), cwd='/')
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/usr/lib/python2.7/subprocess.py", line 574, in check_output
2018-03-09 15:34:03 DEBUG cluster-relation-changed raise CalledProcessError(retcode, cmd, output=output)
2018-03-09 15:34:03 DEBUG cluster-relation-changed subprocess.CalledProcessError: Command '['unison', '-auto', '-batch=true', '-confirmbigdel=false', '-fastcheck=true', '-group=false', '-owner=false', '-prefer=newer', '-times=true', '/var/lib/keystone/juju_sync/juju-ssl-sync.tar', u'ssh://juju_keystone@10.243.8.166//var/lib/keystone/juju_sync/juju-ssl-sync.tar']' returned non-zero exit status 3
2018-03-09 15:34:03 ERROR juju.worker.uniter.operation runhook.go:113 hook "cluster-relation-changed" failed: exit status 1

...

and:

2018-03-09 15:51:17 INFO juju-log cluster:22: Syncing local path /var/lib/keystone/juju_sync/juju-ssl-sync.tar to juju_keystone@10.243.8.166:/var/lib/keystone/juju_sync/juju-ssl-sync.tar
2018-03-09 15:51:17 DEBUG cluster-relation-changed Contacting server...
2018-03-09 15:53:24 DEBUG cluster-relation-changed ssh: connect to host 10.243.8.166 port 22: Connection timed out
2018-03-09 15:53:24 DEBUG cluster-relation-changed Fatal error: Lost connection with the server