autopkgtest 'bonds' fails test test_bond_resend_igmp

[Dan Streetman]

[impact]

autopkgtest 'bonds' frequently fails its 'test_bond_resend_igmp' test

test_bond_resend_igmp (__main__.TestNetworkd) ... ERROR

[test case]

check autopkgtest logs, e.g.
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-eoan/eoan/amd64/n/netplan.io/20200427_185418_ed3e7@/log.gz

[regression potential]

tbd

[discussion]

======================================================================
ERROR: test_bond_resend_igmp (__main__.TestNetworkd)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/autopkgtest.HahtWY/build.cUE/src/tests/integration/bonds.py", line 291, in test_bond_resend_igmp
    self.generate_and_settle([self.dev_e_client, self.dev_e2_client, 'mybond'])
  File "/tmp/autopkgtest.HahtWY/build.cUE/src/tests/integration/base.py", line 300, in generate_and_settle
    out = subprocess.check_output(['netplan', 'apply'], stderr=subprocess.STDOUT, universal_newlines=True)
  File "/usr/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['netplan', 'apply']' returned non-zero exit status 1.

[Bryce Harrington (bryce)]

Unfortunately the test doesn't indicate why netplan apply failed.

Maybe something like this might give more info?

diff --git a/tests/integration/base.py b/tests/integration/base.py
index 5042bf4..c8902ca 100644
--- a/tests/integration/base.py
+++ b/tests/integration/base.py
@@ -297,7 +297,12 @@ class IntegrationTestsBase(unittest.TestCase):
         '''Generate config, launch and settle NM and networkd'''

         # regenerate netplan config
- out = subprocess.check_output(['netplan', 'apply'], stderr=subprocess.STDOUT, universal_newlines=True)
+ try:
+ out = subprocess.check_output(['netplan', 'apply'], stderr=subprocess.STDOUT, universal_newlines=True)
+ except subprocess.CalledProcessError as e:
+ sys.stderr.write('%d: %s\n' %(e.returncode, e.output))
+ self.fail('could not regenerate netplan config')
+
         if 'Run \'systemctl daemon-reload\' to reload units.' in out:
             self.fail('systemd units changed without reload')
         # start NM so that we can verify that it does not manage anything

[Lukas Märdian (slyon)]

Thanks, I did do some similar debugging on my local machine.
'netplan apply' fails because 'networkctl reload' returns 1 due to a systemd-networkd bug/race condition that only seems to happen in some situations, especially if the machine is loaded.

https://github.com/systemd/systemd/issues/18108

It should be fixed in netplan.io 0.103-0ubuntu3 where I reverted the new functionality of netplan to make use of 'networkctl reload' – we can re-enable this once systemd v249 hits the archive.