I know you've destroyed and are about to redeploy with latest wares, and the following relates to the initial deploy referenced in this bug. But pasting observations thus-far:
The traceback in the uploaded file is completely different than that shown in the bug description. While it may be related, I'd like to see the unit log and keystone deamon logs for the same timeframe matching the traceback from the bug description (or a repro of that).
Also, I'm not able to determine what the .166 address is from the model status or bundles pastes, but this is a problem:
ssh: connect to host 10.243.8.166 port 22: Connection timed out
This could be a network or MTU issue. If the issue it recurs with port 22 connect timeouts, please confirm switchport MTUs are high enough for NIC MTUs and container interface MTUs.
2018-03-09 15:34:03 INFO juju-log cluster:22: Error syncing remote files
2018-03-09 15:34:03 DEBUG cluster-relation-changed Traceback (most recent call last):
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-changed", line 880, in <module>
2018-03-09 15:34:03 DEBUG cluster-relation-changed main()
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-changed", line 873, in main
2018-03-09 15:34:03 DEBUG cluster-relation-changed hooks.execute(sys.argv)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/hookenv.py", line 800, in execute
2018-03-09 15:34:03 DEBUG cluster-relation-changed self._hooks[hook_name]()
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1449, in wrapped_f
2018-03-09 15:34:03 DEBUG cluster-relation-changed restart_functions)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/host.py", line 730, in restart_on_change_helper
2018-03-09 15:34:03 DEBUG cluster-relation-changed r = lambda_f()
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1448, in <lambda>
2018-03-09 15:34:03 DEBUG cluster-relation-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1650, in _inner_update_certs_if_available
2018-03-09 15:34:03 DEBUG cluster-relation-changed return f(*args, **kwargs)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-changed", line 618, in cluster_changed
2018-03-09 15:34:03 DEBUG cluster-relation-changed update_all_identity_relation_units_force_sync()
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1824, in inner_synchronize_ca_if_changed2
2018-03-09 15:34:03 DEBUG cluster-relation-changed peer_settings = synchronize_ca(fatal=fatal)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1724, in synchronize_ca
2018-03-09 15:34:03 DEBUG cluster-relation-changed synced_units = unison_sync([SSL_SYNC_ARCHIVE, SYNC_FLAGS_DIR])
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
2018-03-09 15:34:03 DEBUG cluster-relation-changed return f(*args, **kwargs)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1445, in unison_sync
2018-03-09 15:34:03 DEBUG cluster-relation-changed fatal=True)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/unison/__init__.py", line 314, in sync_to_peers
2018-03-09 15:34:03 DEBUG cluster-relation-changed sync_to_peer(host, user, paths, verbose, cmd, gid, fatal)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/unison/__init__.py", line 300, in sync_to_peer
2018-03-09 15:34:03 DEBUG cluster-relation-changed sync_path_to_host(p, host, user, verbose, cmd, gid, fatal)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/unison/__init__.py", line 285, in sync_path_to_host
2018-03-09 15:34:03 DEBUG cluster-relation-changed run_as_user(user, cmd, gid)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/unison/__init__.py", line 239, in run_as_user
2018-03-09 15:34:03 DEBUG cluster-relation-changed return check_output(cmd, preexec_fn=_run_as_user(user, gid), cwd='/')
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/usr/lib/python2.7/subprocess.py", line 574, in check_output
2018-03-09 15:34:03 DEBUG cluster-relation-changed raise CalledProcessError(retcode, cmd, output=output)
2018-03-09 15:34:03 DEBUG cluster-relation-changed subprocess.CalledProcessError: Command '['unison', '-auto', '-batch=true', '-confirmbigdel=false', '-fastcheck=true', '-group=false', '-owner=false', '-prefer=newer', '-times=true', '/var/lib/keystone/juju_sync/juju-ssl-sync.tar', u'ssh://juju_keystone@10.243.8.166//var/lib/keystone/juju_sync/juju-ssl-sync.tar']' returned non-zero exit status 3
2018-03-09 15:34:03 ERROR juju.worker.uniter.operation runhook.go:113 hook "cluster-relation-changed" failed: exit status 1
...
and:
2018-03-09 15:51:17 INFO juju-log cluster:22: Syncing local path /var/lib/keystone/juju_sync/juju-ssl-sync.tar to juju_keystone@10.243.8.166:/var/lib/keystone/juju_sync/juju-ssl-sync.tar
2018-03-09 15:51:17 DEBUG cluster-relation-changed Contacting server...
2018-03-09 15:53:24 DEBUG cluster-relation-changed ssh: connect to host 10.243.8.166 port 22: Connection timed out
2018-03-09 15:53:24 DEBUG cluster-relation-changed Fatal error: Lost connection with the server
I know you've destroyed and are about to redeploy with latest wares, and the following relates to the initial deploy referenced in this bug. But pasting observations thus-far:
The traceback in the uploaded file is completely different than that shown in the bug description. While it may be related, I'd like to see the unit log and keystone deamon logs for the same timeframe matching the traceback from the bug description (or a repro of that).
Also, I'm not able to determine what the .166 address is from the model status or bundles pastes, but this is a problem:
ssh: connect to host 10.243.8.166 port 22: Connection timed out
This could be a network or MTU issue. If the issue it recurs with port 22 connect timeouts, please confirm switchport MTUs are high enough for NIC MTUs and container interface MTUs.
https:/ /pastebin. canonical. com/p/jGGZXbH2m 6/
and
https:/ /pastebin. canonical. com/p/zP9xZZcGp s/
...
2018-03-09 15:34:03 INFO juju-log cluster:22: Error syncing remote files relation- changed Traceback (most recent call last): relation- changed File "/var/lib/ juju/agents/ unit-keystone- 0/charm/ hooks/cluster- relation- changed" , line 880, in <module> relation- changed main() relation- changed File "/var/lib/ juju/agents/ unit-keystone- 0/charm/ hooks/cluster- relation- changed" , line 873, in main relation- changed hooks.execute( sys.argv) relation- changed File "/var/lib/ juju/agents/ unit-keystone- 0/charm/ hooks/charmhelp ers/core/ hookenv. py", line 800, in execute relation- changed self._hooks[ hook_name] () relation- changed File "/var/lib/ juju/agents/ unit-keystone- 0/charm/ hooks/charmhelp ers/contrib/ openstack/ utils.py" , line 1449, in wrapped_f relation- changed restart_functions) relation- changed File "/var/lib/ juju/agents/ unit-keystone- 0/charm/ hooks/charmhelp ers/core/ host.py" , line 730, in restart_ on_change_ helper relation- changed r = lambda_f() relation- changed File "/var/lib/ juju/agents/ unit-keystone- 0/charm/ hooks/charmhelp ers/contrib/ openstack/ utils.py" , line 1448, in <lambda> relation- changed (lambda: f(*args, **kwargs)), restart_map, stopstart, relation- changed File "/var/lib/ juju/agents/ unit-keystone- 0/charm/ hooks/keystone_ utils.py" , line 1650, in _inner_ update_ certs_if_ available relation- changed return f(*args, **kwargs) relation- changed File "/var/lib/ juju/agents/ unit-keystone- 0/charm/ hooks/cluster- relation- changed" , line 618, in cluster_changed relation- changed update_ all_identity_ relation_ units_force_ sync() relation- changed File "/var/lib/ juju/agents/ unit-keystone- 0/charm/ hooks/keystone_ utils.py" , line 1824, in inner_synchroni ze_ca_if_ changed2 relation- changed peer_settings = synchronize_ ca(fatal= fatal) relation- changed File "/var/lib/ juju/agents/ unit-keystone- 0/charm/ hooks/keystone_ utils.py" , line 1724, in synchronize_ca relation- changed synced_units = unison_ sync([SSL_ SYNC_ARCHIVE, SYNC_FLAGS_DIR]) relation- changed File "/var/lib/ juju/agents/ unit-keystone- 0/charm/ hooks/charmhelp ers/core/ decorators. py", line 40, in _retry_ on_exception_ inner_2 relation- changed return f(*args, **kwargs) relation- changed File "/var/lib/ juju/agents/ unit-keystone- 0/charm/ hooks/keystone_ utils.py" , line 1445, in unison_sync relation- changed fatal=True) relation- changed File "/var/lib/ juju/agents/ unit-keystone- 0/charm/ hooks/charmhelp ers/contrib/ unison/ __init_ _.py", line 314, in sync_to_peers relation- changed sync_to_peer(host, user, paths, verbose, cmd, gid, fatal) relation- changed File "/var/lib/ juju/agents/ unit-keystone- 0/charm/ hooks/charmhelp ers/contrib/ unison/ __init_ _.py", line 300, in sync_to_peer relation- changed sync_path_ to_host( p, host, user, verbose, cmd, gid, fatal) relation- changed File "/var/lib/ juju/agents/ unit-keystone- 0/charm/ hooks/charmhelp ers/contrib/ unison/ __init_ _.py", line 285, in sync_path_to_host relation- changed run_as_user(user, cmd, gid) relation- changed File "/var/lib/ juju/agents/ unit-keystone- 0/charm/ hooks/charmhelp ers/contrib/ unison/ __init_ _.py", line 239, in run_as_user relation- changed return check_output(cmd, preexec_ fn=_run_ as_user( user, gid), cwd='/') relation- changed File "/usr/lib/ python2. 7/subprocess. py", line 574, in check_output relation- changed raise CalledProcessEr ror(retcode, cmd, output=output) relation- changed subprocess. CalledProcessEr ror: Command '['unison', '-auto', '-batch=true', '-confirmbigdel =false' , '-fastcheck=true', '-group=false', '-owner=false', '-prefer=newer', '-times=true', '/var/lib/ keystone/ juju_sync/ juju-ssl- sync.tar' , u'ssh:/ /juju_keystone@ 10.243. 8.166// var/lib/ keystone/ juju_sync/ juju-ssl- sync.tar' ]' returned non-zero exit status 3 uniter. operation runhook.go:113 hook "cluster- relation- changed" failed: exit status 1
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 DEBUG cluster-
2018-03-09 15:34:03 ERROR juju.worker.
...
and:
2018-03-09 15:51:17 INFO juju-log cluster:22: Syncing local path /var/lib/ keystone/ juju_sync/ juju-ssl- sync.tar to juju_keystone@ 10.243. 8.166:/ var/lib/ keystone/ juju_sync/ juju-ssl- sync.tar relation- changed Contacting server... relation- changed ssh: connect to host 10.243.8.166 port 22: Connection timed out relation- changed Fatal error: Lost connection with the server
2018-03-09 15:51:17 DEBUG cluster-
2018-03-09 15:53:24 DEBUG cluster-
2018-03-09 15:53:24 DEBUG cluster-