controller-1 log did not collected.
automation log shows that "server list" cmd hung for 10 mins.
[2019-06-21 12:58:53,140] 268 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server list --a' [2019-06-21 13:08:53,248] 358 WARNING MainThread ssh.expect :: No match found for ['.*controller\\-[01][:| ].*\\$ ']. expect timeout. [2019-06-21 13:08:53,249] 691 DEBUG MainThread ssh.send_control:: Sending ctrl+c [2019-06-21 13:08:53,336] 387 DEBUG MainThread ssh.expect :: Output: ^CTraceback (most recent call last): File "/usr/bin/openstack", line 10, in <module> sys.exit(main()) File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 210, in main return OpenStackShell().run(argv) File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run ret_val = super(OpenStackShell, self).run(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 281, in run result = self.run_subcommand(remainder) File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 175, in run_subcommand ret_value = super(OpenStackShell, self).run_subcommand(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/display.py", line 116, in run column_names, data = self.take_action(parsed_args) File "/usr/lib/python2.7/site-packages/openstackclient/compute/v2/server.py", line 1275, in take_action limit=parsed_args.limit) File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 892, in list "servers") File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 254, in _list resp, body = self.api.client.get(url) File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 375, in get return self.request(url, 'GET', **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 72, in request **kwargs) File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 534, in request resp = super(LegacyJsonAdapter, self).request(*args, **kwargs) File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 237, in request return self.session.request(url, method, **kwargs) File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 835, in request resp = send(**kwargs) File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 926, in _send_request resp = self.session.request(method, url, **kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 379, in _make_request httplib_response = conn.getresponse(buffering=True) File "/usr/lib64/python2.7/httplib.py", line 1113, in getresponse response.begin() File "/usr/lib64/python2.7/httplib.py", line 444, in begin version, status, reason = self._read_status() File "/usr/lib64/python2.7/httplib.py", line 400, in _read_status line = self.fp.readline(_MAXLINE + 1) File "/usr/lib64/python2.7/socket.py", line 476, in readline data = self._sock.recv(self._rbufsize) KeyboardInterrupt controller-1:~$ [2019-06-21 13:08:53,336] 308 DEBUG MainThread ssh.flush :: Buffer is flushed by reading out the rest of the output [2019-06-21 13:08:53,336] 439 WARNING MainThread ssh.exec_cmd:: Timeout exceeded. <pexpect.pxssh.pxssh object at 0x7f3a7af1fc18> command: /usr/bin/ssh args: ['/usr/bin/ssh', '-o', 'RSAAuthentication=no', '-o', 'PubkeyAuthentication=no', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/dev/null', '-l', 'sysadmin', '128.224.150.254'] searcher: None buffer (last 100 chars): 't --os-identity-api-version 3 - \r-os-interface internal --os-region-name RegionOne server list --a\r\n' before (last 100 chars): 't --os-identity-api-version 3 - \r-os-interface internal --os-region-name RegionOne server list --a\r\n' after: <class 'pexpect.exceptions.TIMEOUT'> match: None match_index: None exitstatus: None flag_eof: False pid: 14030 child_fd: 17 closed: False timeout: 30 delimiter: <class 'pexpect.exceptions.EOF'> logfile: <_io.TextIOWrapper name='/sandbox/AUTOMATION_LOGS/wp_1_2/201906210650/ssh_wp_1_2.log' mode='a+' encoding='UTF-8'> logfile_read: None logfile_send: None maxread: 100000 ignorecase: False searchwindowsize: None delaybeforesend: 0.05 delayafterclose: 0.1 delayafterterminate: 0.1 [2019-06-21 13:08:53,336] 268 DEBUG MainThread ssh.send :: Send 'echo $?'
controller-1 log did not collected.
automation log shows that "server list" cmd hung for 10 mins.
[2019-06-21 12:58:53,140] 268 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http:// keystone. openstack. svc.cluster. local/v3 --os-user- domain- name Default --os-project- domain- name Default --os-identity- api-version 3 --os-interface internal --os-region-name RegionOne server list --a' \-[01][ :| ].*\\$ ']. openstack" , line 10, in <module> exit(main( )) python2. 7/site- packages/ openstackclient /shell. py", line 210, in main ).run(argv) python2. 7/site- packages/ osc_lib/ shell.py" , line 135, in run Shell, self).run(argv) python2. 7/site- packages/ cliff/app. py", line 281, in run subcommand( remainder) python2. 7/site- packages/ osc_lib/ shell.py" , line 175, in run_subcommand Shell, self).run_ subcommand( argv) python2. 7/site- packages/ cliff/app. py", line 402, in run_subcommand parsed_ args) python2. 7/site- packages/ osc_lib/ command/ command. py", line 41, in run parsed_ args) python2. 7/site- packages/ cliff/display. py", line 116, in run action( parsed_ args) python2. 7/site- packages/ openstackclient /compute/ v2/server. py", line 1275, in take_action parsed_ args.limit) python2. 7/site- packages/ novaclient/ v2/servers. py", line 892, in list python2. 7/site- packages/ novaclient/ base.py" , line 254, in _list client. get(url) python2. 7/site- packages/ keystoneauth1/ adapter. py", line 375, in get python2. 7/site- packages/ novaclient/ client. py", line 72, in request python2. 7/site- packages/ keystoneauth1/ adapter. py", line 534, in request nAdapter, self).request( *args, **kwargs) python2. 7/site- packages/ keystoneauth1/ adapter. py", line 237, in request request( url, method, **kwargs) python2. 7/site- packages/ keystoneauth1/ session. py", line 835, in request python2. 7/site- packages/ keystoneauth1/ session. py", line 926, in _send_request request( method, url, **kwargs) python2. 7/site- packages/ requests/ sessions. py", line 533, in request python2. 7/site- packages/ requests/ sessions. py", line 646, in send send(request, **kwargs) python2. 7/site- packages/ requests/ adapters. py", line 449, in send python2. 7/site- packages/ urllib3/ connectionpool. py", line 600, in urlopen chunked) python2. 7/site- packages/ urllib3/ connectionpool. py", line 379, in _make_request response = conn.getrespons e(buffering= True) python2. 7/httplib. py", line 1113, in getresponse begin() python2. 7/httplib. py", line 444, in begin python2. 7/httplib. py", line 400, in _read_status readline( _MAXLINE + 1) python2. 7/socket. py", line 476, in readline recv(self. _rbufsize) pxssh.pxssh object at 0x7f3a7af1fc18> ion=no' , '-o', 'PubkeyAuthenti cation= no', '-o', 'StrictHostKeyC hecking= no', '-o', 'UserKnownHosts File=/dev/ null', '-l', 'sysadmin', '128.224.150.254'] api-version 3 - \r-os-interface internal --os-region-name RegionOne server list --a\r\n' api-version 3 - \r-os-interface internal --os-region-name RegionOne server list --a\r\n' exceptions. TIMEOUT' > exceptions. EOF'> sandbox/ AUTOMATION_ LOGS/wp_ 1_2/20190621065 0/ssh_wp_ 1_2.log' mode='a+' encoding='UTF-8'> nate: 0.1
[2019-06-21 13:08:53,248] 358 WARNING MainThread ssh.expect :: No match found for ['.*controller\
expect timeout.
[2019-06-21 13:08:53,249] 691 DEBUG MainThread ssh.send_control:: Sending ctrl+c
[2019-06-21 13:08:53,336] 387 DEBUG MainThread ssh.expect :: Output:
^CTraceback (most recent call last):
File "/usr/bin/
sys.
File "/usr/lib/
return OpenStackShell(
File "/usr/lib/
ret_val = super(OpenStack
File "/usr/lib/
result = self.run_
File "/usr/lib/
ret_value = super(OpenStack
File "/usr/lib/
result = cmd.run(
File "/usr/lib/
return super(Command, self).run(
File "/usr/lib/
column_names, data = self.take_
File "/usr/lib/
limit=
File "/usr/lib/
"servers")
File "/usr/lib/
resp, body = self.api.
File "/usr/lib/
return self.request(url, 'GET', **kwargs)
File "/usr/lib/
**kwargs)
File "/usr/lib/
resp = super(LegacyJso
File "/usr/lib/
return self.session.
File "/usr/lib/
resp = send(**kwargs)
File "/usr/lib/
resp = self.session.
File "/usr/lib/
resp = self.send(prep, **send_kwargs)
File "/usr/lib/
r = adapter.
File "/usr/lib/
timeout=timeout
File "/usr/lib/
chunked=
File "/usr/lib/
httplib_
File "/usr/lib64/
response.
File "/usr/lib64/
version, status, reason = self._read_status()
File "/usr/lib64/
line = self.fp.
File "/usr/lib64/
data = self._sock.
KeyboardInterrupt
controller-1:~$
[2019-06-21 13:08:53,336] 308 DEBUG MainThread ssh.flush :: Buffer is flushed by reading out the rest of the output
[2019-06-21 13:08:53,336] 439 WARNING MainThread ssh.exec_cmd:: Timeout exceeded.
<pexpect.
command: /usr/bin/ssh
args: ['/usr/bin/ssh', '-o', 'RSAAuthenticat
searcher: None
buffer (last 100 chars): 't --os-identity-
before (last 100 chars): 't --os-identity-
after: <class 'pexpect.
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 14030
child_fd: 17
closed: False
timeout: 30
delimiter: <class 'pexpect.
logfile: <_io.TextIOWrapper name='/
logfile_read: None
logfile_send: None
maxread: 100000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayaftertermi
[2019-06-21 13:08:53,336] 268 DEBUG MainThread ssh.send :: Send 'echo $?'