Comment 3 for bug 1833730

Revision history for this message
Peng Peng (ppeng) wrote : Re: "openstack server list" cmd not working after standby controller reboot

controller-1 log did not collected.

automation log shows that "server list" cmd hung for 10 mins.

[2019-06-21 12:58:53,140] 268 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server list --a'
[2019-06-21 13:08:53,248] 358 WARNING MainThread ssh.expect :: No match found for ['.*controller\\-[01][:| ].*\\$ '].
expect timeout.
[2019-06-21 13:08:53,249] 691 DEBUG MainThread ssh.send_control:: Sending ctrl+c
[2019-06-21 13:08:53,336] 387 DEBUG MainThread ssh.expect :: Output:
^CTraceback (most recent call last):
  File "/usr/bin/openstack", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 210, in main
    return OpenStackShell().run(argv)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 281, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 175, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/display.py", line 116, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/openstackclient/compute/v2/server.py", line 1275, in take_action
    limit=parsed_args.limit)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 892, in list
    "servers")
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 254, in _list
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 375, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 72, in request
    **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 534, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 237, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 835, in request
    resp = send(**kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 926, in _send_request
    resp = self.session.request(method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 379, in _make_request
    httplib_response = conn.getresponse(buffering=True)
  File "/usr/lib64/python2.7/httplib.py", line 1113, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 444, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 400, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib64/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
KeyboardInterrupt
controller-1:~$
[2019-06-21 13:08:53,336] 308 DEBUG MainThread ssh.flush :: Buffer is flushed by reading out the rest of the output
[2019-06-21 13:08:53,336] 439 WARNING MainThread ssh.exec_cmd:: Timeout exceeded.
<pexpect.pxssh.pxssh object at 0x7f3a7af1fc18>
command: /usr/bin/ssh
args: ['/usr/bin/ssh', '-o', 'RSAAuthentication=no', '-o', 'PubkeyAuthentication=no', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/dev/null', '-l', 'sysadmin', '128.224.150.254']
searcher: None
buffer (last 100 chars): 't --os-identity-api-version 3 - \r-os-interface internal --os-region-name RegionOne server list --a\r\n'
before (last 100 chars): 't --os-identity-api-version 3 - \r-os-interface internal --os-region-name RegionOne server list --a\r\n'
after: <class 'pexpect.exceptions.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 14030
child_fd: 17
closed: False
timeout: 30
delimiter: <class 'pexpect.exceptions.EOF'>
logfile: <_io.TextIOWrapper name='/sandbox/AUTOMATION_LOGS/wp_1_2/201906210650/ssh_wp_1_2.log' mode='a+' encoding='UTF-8'>
logfile_read: None
logfile_send: None
maxread: 100000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
[2019-06-21 13:08:53,336] 268 DEBUG MainThread ssh.send :: Send 'echo $?'