container based tests can fail with "RecursionError: maximum recursion depth exceeded in comparison"

[Iain Lane]

I think this is probably a bug in autopkgtest rather than -cloud.

Seems to happen when the controller machine has trouble talking to the instance via the lxd remote, the tests fail like:

Traceback (most recent call last):
  File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 739, in mainloop
    command()
  File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 668, in command
    r = f(c, ce)
  File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 363, in cmd_reboot
    caller.hook_wait_reboot()
  File "/home/ubuntu/autopkgtest/virt/autopkgtest-virt-lxd", line 231, in hook_wait_reboot
    wait_booted()
  File "/home/ubuntu/autopkgtest/virt/autopkgtest-virt-lxd", line 104, in wait_booted
    VirtSubproc.check_exec(['lxc', 'exec', container_name, '--', 'sh', '-ec', r'if [ -d /run/systemd/system ]; then systemctl start network-online.target; else while ps -ef | grep -q "/etc/init\.d/rc"; do sleep 1; done; fi'], timeout=60)
  File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 183, in check_exec
    stdout=stdout, stderr=subprocess.PIPE)
  File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 144, in execute_timeout
    (out, err) = sp.communicate(instr)
  File "/usr/lib/python3.5/subprocess.py", line 1062, in communicate
    stderr = self.stderr.read()
  File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 64, in alarm_handler
    raise Timeout()
VirtSubproc.Timeout
autopkgtest [00:27:06]: ERROR: unexpected error:
Traceback (most recent call last):
  File "/home/ubuntu/autopkgtest/lib/adt_testbed.py", line 475, in send
    self.sp.stdin.flush()
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/autopkgtest/lib/adt_testbed.py", line 475, in send
    self.sp.stdin.flush()
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/autopkgtest/lib/adt_testbed.py", line 475, in send
    self.sp.stdin.flush()
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/autopkgtest/lib/adt_testbed.py", line 475, in send
    self.sp.stdin.flush()
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 838, in main
    testbed.open()
  File "/home/ubuntu/autopkgtest/lib/adt_testbed.py", line 152, in open
    self._opened(pl)
  File "/home/ubuntu/autopkgtest/lib/adt_testbed.py", line 246, in _opened
    self.run_setup_commands()
  File "/home/ubuntu/autopkgtest/lib/adt_testbed.py", line 411, in run_setup_commands
    self.reboot()
  File "/home/ubuntu/autopkgtest/lib/adt_testbed.py", line 296, in reboot
    self.command('reboot', prepare_only and ('prepare-only', ) or ())
  File "/home/ubuntu/autopkgtest/lib/adt_testbed.py", line 522, in command
    ll = self.expect('ok', nresults)
  File "/home/ubuntu/autopkgtest/lib/adt_testbed.py", line 487, in expect
    self.command('auxverb_debug_fail')

and then this repeats until we hit the RecursionError. We should fail cleanly.

Then we return an "unknown" result, which is annoying because they need to be retried. For that case I'll add a string to the worker to get them retried at least. Then we get three attempts before tests die with this - and if that happens then the remote is probably in trouble.