checkbox-remote crash after resume from remote master

[Ray Chen]

Steps to reproduced
1. Run checkbox-cli slave and checkbox-cli remote localhost on DUT (18.04)
2. After few tests, Hit Ctrl-C twice to disconnect form slave
3. Resume checkbox-cli master DUT_IP on another machine (16.04)

got below traceback

u@Ray-ConsoleWatcher-Inspiron-5477-AIO:~$ checkbox-cli master 10.101.46.163
Connecting to 10.101.46.163:18871. Timeout: 30s
rejoined session. Running job (50/434): com.canonical.certification::audio/channels
Enter sudo password:

-------------------------------[ audio/channels ]-------------------------------
ID: com.canonical.certification::audio/channels
Category: Audio tests

                        Application Malfunction Detected

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/guacamole/core.py", line 248, in eat
    return self._dispatch()
  File "/usr/lib/python3/dist-packages/guacamole/core.py", line 298, in _dispatch
    result = ingredient.dispatch(self.context)
  File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line 152, in dispatch
    return self._dispatch(context, 0)
  File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line 179, in _dispatch
    return self._dispatch_None(context, level, retval, command)
  File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line 208, in _dispatch_None
    return self._dispatch(context, level + 1)
  File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line 165, in _dispatch
    retval = command.invoked(context)
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/remote.py", line 170, in invoked
    self.connect_and_run(ctx.args.host, port)
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/remote.py", line 224, in connect_and_run
    }[state]()
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/remote.py", line 348, in wait_and_continue
    self.continue_session()
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/remote.py", line 352, in continue_session
    self.run_jobs(todo)
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/remote.py", line 364, in run_jobs
    for interaction in self.sa.run_job(job['id']):
  File "/usr/lib/python3/dist-packages/plainbox/vendor/rpyc/core/netref.py", line 199, in __call__
    return syncreq(_self, consts.HANDLE_CALL, args, kwargs)
  File "/usr/lib/python3/dist-packages/plainbox/vendor/rpyc/core/netref.py", line 72, in syncreq
    return conn.sync_request(handler, oid, *args)
  File "/usr/lib/python3/dist-packages/plainbox/vendor/rpyc/core/protocol.py", line 523, in sync_request
    raise obj
_get_exception_class.<locals>.Derived: expected ('testsselected',), is running

========= Remote Traceback (1) =========
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/plainbox/vendor/rpyc/core/protocol.py", line 347, in _dispatch_request
    res = self._HANDLERS[handler](self, *args)
  File "/usr/lib/python3/dist-packages/plainbox/vendor/rpyc/core/protocol.py", line 624, in _handle_call
    return self._local_objects[oid](*args, **dict(kwargs))
  File "/usr/lib/python3/dist-packages/plainbox/impl/session/remote_assistant.py", line 169, in fun
    "expected %s, is %s" % (states, self._state))
AssertionError: expected ('testsselected',), is running

  Please report a bug including the information from the paragraph above. To
  report the bug visit https://bugs.launchpad.net/checkbox-ng/+filebug

  We are sorry for the inconvenience!

================================================================================
The DUT are 18.04 and remote master are 16.04

[Ray Chen (ray.chen)]

Trace from Slave (18.04)

u@u-Precision-5530:~$ checkbox-cli slave
local script com.canonical.certification::module returned invalid RFC822 data: Unexpected non-empty line: 'depend\n' (line 264)
Exception in thread Thread-71:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3/dist-packages/plainbox/impl/session/remote_assistant.py", line 116, in run
    self._job_id, self._sa.buffered_ui, False)
  File "/usr/lib/python3/dist-packages/plainbox/impl/decorators.py", line 142, in wrapper
    raise exc
  File "/usr/lib/python3/dist-packages/plainbox/impl/decorators.py", line 136, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/plainbox/impl/session/assistant.py", line 1296, in run_job
    UsageExpectation.of(self).enforce()
  File "/usr/lib/python3/dist-packages/plainbox/impl/developer.py", line 222, in enforce
    raise UnexpectedMethodCall(self.cls, fn_name, allowed_pairs)
plainbox.impl.developer.UnexpectedMethodCall:
Uh, oh...

You are not expected to call SessionAssistant.run_job() at this time.

If you see this message then there is a bug somewhere in your code. We are
sorry for this. Perhaps the documentation is flawed, incomplete or confusing.
Please reach out to us if this happens more often than you'd like.

The set of allowed calls, at this time, is:

 - call SessionAssistant.export_to_file() to to export the results to a file.
 - call SessionAssistant.export_to_stream() to to export the results to a stream.
 - call SessionAssistant.export_to_transport() to to export the results and send them.
 - call SessionAssistant.filter_jobs_by_categories() to to select the jobs that match particular category.
 - call SessionAssistant.finalize_session() to to mark the session as complete.
 - call SessionAssistant.finish_bootstrap() to to finish bootstrapping.
 - call SessionAssistant.get_category() to to access the definition of ant category.
 - call SessionAssistant.get_dynamic_todo_list() to to see what is yet to be executed.
 - call SessionAssistant.get_job() to to access the definition of any job.
 - call SessionAssistant.get_job_state() to to access the state of any job.
 - call SessionAssistant.get_mandatory_jobs() to to get all mandatory job ids.
 - call SessionAssistant.get_participating_categories() to to access participating categories.
 - call SessionAssistant.get_session_dir() to to get the path where current session isstored.
 - call SessionAssistant.get_session_id() to to get the id of currently running session.
 - call SessionAssistant.get_static_todo_list() to to see what is meant to be executed.
 - call SessionAssistant.get_test_plan() to to access the definition of any test plan.
 - call SessionAssistant.hand_pick_jobs() to to generate new selection and use it.
 - call SessionAssistant.remove_all_filters() to to remove all filters.
 - call SessionAssistant.use_alternate_selection() to to change the selection.
 - call SessionAssistant.use_job_result() to remember the result of this job.

Refer to the documentation of SessionAssistant for details.
    TIP: python -m pydoc plainbox.impl.session.assistan...

[Maciej Kisielewski (kissiel)]

This got fixed in one of the two previous releases with major reworks done, so marking this as 'fix released'