Checkbox remote master traceback when 2 instances connected to one slave

[Jeff Lane ]

for fun, I kicked off checkbox remote slave on a SUT and then connected from my desktop using checkbox remote master (Connection 1) and started a short testplan.

Then from a different machine (Connection 2) I also connected to the slave to see what would happen.

This seems to have caused the checkbox master on Connection 1 to traceback with an unhandled exception that doesn't provide any helpful info:

--------------------------------------------------------------------------------
Outcome: job passed
-----------------------------[ Running job 2 / 74 ]-----------------------------
---------[ Attaches json dumps of installed dkms package information. ]---------
ID: com.canonical.certification::dkms_info_json
Category: Informational tests
--------------------------------------------------------------------------------

                        Application Malfunction Detected

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/guacamole/core.py", line 248, in eat
    return self._dispatch()
  File "/usr/lib/python3/dist-packages/guacamole/core.py", line 298, in _dispatch
    result = ingredient.dispatch(self.context)
  File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line 152, in dispatch
    return self._dispatch(context, 0)
  File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line 179, in _dispatch
    return self._dispatch_None(context, level, retval, command)
  File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line 208, in _dispatch_None
    return self._dispatch(context, level + 1)
  File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line 165, in _dispatch
    retval = command.invoked(context)
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/master.py", line 141, in invoked
    self.connect_and_run(ctx.args.host, port)
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/master.py", line 198, in connect_and_run
    }[state]()
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/master.py", line 226, in new_session
    self.select_jobs(self.jobs)
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/master.py", line 280, in select_jobs
    self.run_jobs()
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/master.py", line 369, in run_jobs
    self._run_jobs(jobs_repr, total_num)
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/master.py", line 481, in _run_jobs
    for interaction in self.sa.run_job(job['id']):
  File "/usr/lib/python3/dist-packages/plainbox/vendor/rpyc/core/netref.py", line 199, in __call__
    return syncreq(_self, consts.HANDLE_CALL, args, kwargs)
  File "/usr/lib/python3/dist-packages/plainbox/vendor/rpyc/core/netref.py", line 75, in syncreq
    return conn.sync_request(handler, proxy, *args)
  File "/usr/lib/python3/dist-packages/plainbox/vendor/rpyc/core/protocol.py", line 471, in sync_request
    return self.async_request(handler, *args, timeout=timeout).value
  File "/usr/lib/python3/dist-packages/plainbox/vendor/rpyc/core/async_.py", line 97, in value
    raise self._obj
_get_exception_class.<locals>.Derived: expected ('testsselected',), is running

========= Remote Traceback (1) =========
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/plainbox/vendor/rpyc/core/protocol.py", line 329, in _dispatch_request
    res = self._HANDLERS[handler](self, *args)
  File "/usr/lib/python3/dist-packages/plainbox/vendor/rpyc/core/protocol.py", line 590, in _handle_call
    return obj(*args, **dict(kwargs))
  File "/usr/lib/python3/dist-packages/plainbox/impl/session/remote_assistant.py", line 165, in fun
    "expected %s, is %s" % (states, self._state))
AssertionError: expected ('testsselected',), is running

  Please report a bug including the information from the paragraph above. To
  report the bug visit https://bugs.launchpad.net/checkbox-ng/+filebug

  We are sorry for the inconvenience!

Meanwhile, the output on Connection 2 continued all the way to the end of testing.

So to recreate this:
1: start checkbox-cli slave on SUT
2: from terminal 1, start a test using checkbox-cli master IP_ADDR /path/to/launcher
3: while test is running, on terminal 2, start checkbox-cli mater IP_ADDR
4: wait for it...
5: wait for it...
6: checkbox-cli in terminal 1 crashes with that assertion error.

I get that you're not supposed to connect from more than one location at a time, so the remedy here is really more about exception handling so that users get a helpful message like:

"Unable to connect at this time, there is another session already active for this remote SUT"

OR, better would be the ability to disconnect the other session (not sure what the mechanics are behind the remote connection, but something like this:

"There is another active connection to the remote agent on $IPADDRESS"
"Shall I disconnect that session and connect this one? [Y/n]"