rackd fails on CIS-hardened machine with "Failed to update and/or record network interface configuration: Expecting value: line 1 column 1 (char 0)"

[Przemyslaw Hausman]

I'm installing MAAS snap 3.2 on CIS-hardened machine. MAAS installs successfully, but when I go to GUI → Controllers, I can see that there is zero VLANs available for each controller.

In /var/snap/maas/common/log/rackd.log I can see the following critical error:

```
2022-09-16 12:12:35 provisioningserver.utils.services: [critical] Failed to update and/or record network interface configuration: Expecting value: line 1 column 1 (char 0); interfaces: {'bond0': {'type': 'bond', 'mac_address': '62:90:14:00:54:8f', 'links': [], 'enabled': True, 'parents': ['ens4f1np1', 'ens4f0np0', 'ens1f1np1', 'ens1f0np0'], 'source': 'machine-resources', 'monitored': True}, 'bond0.1535': {'type': 'vlan', 'mac_address': '62:90:14:00:54:8f', 'links': [], 'enabled': True, 'parents': ['bond0'], 'source': 'machine-resources', 'vid': 1535, 'monitored': False}, 'bondM': {'type': 'bond', 'mac_address': '66:54:1b:c9:5f:08', 'links': [], 'enabled': True, 'parents': ['eno5', 'eno1'], 'source': 'machine-resources', 'monitored': True}, 'brinternal': {'type': 'bridge', 'mac_address': '62:90:14:00:54:8f', 'links': [{'mode': 'static', 'address': '192.168.30.6/23'}], 'enabled': True, 'parents': ['bond0.1535'], 'source': 'machine-resources', 'monitored': False}, 'broam': {'type': 'bridge', 'mac_address': '66:54:1b:c9:5f:08', 'links': [{'mode': 'static', 'address': '10.46.18.6/23', 'gateway': '10.46.18.1'}, {'mode': 'static', 'address': '10.46.18.52/23', 'gateway': '10.46.18.1'}], 'enabled': True, 'parents': ['bondM'], 'source': 'machine-resources', 'monitored': False}, 'eno1': {'type': 'physical', 'mac_address': 'b4:7a:f1:e0:4d:ce', 'links': [], 'enabled': True, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'eno2': {'type': 'physical', 'mac_address': 'b4:7a:f1:e0:4d:cf', 'links': [], 'enabled': False, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'eno3': {'type': 'physical', 'mac_address': 'b4:7a:f1:e0:4d:d0', 'links': [], 'enabled': False, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'eno4': {'type': 'physical', 'mac_address': 'b4:7a:f1:e0:4d:d1', 'links': [], 'enabled': False, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'eno5': {'type': 'physical', 'mac_address': '98:f2:b3:23:43:e0', 'links': [], 'enabled': True, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'eno6': {'type': 'physical', 'mac_address': '98:f2:b3:23:43:e1', 'links': [], 'enabled': False, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'eno7': {'type': 'physical', 'mac_address': '98:f2:b3:23:43:e2', 'links': [], 'enabled': False, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'eno8': {'type': 'physical', 'mac_address':'98:f2:b3:23:43:e3', 'links': [], 'enabled': False, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'ens1f0np0': {'type': 'physical', 'mac_address': 'f4:03:43:ee:e9:50', 'links': [], 'enabled': True, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'ens1f1np1': {'type': 'physical', 'mac_address': 'f4:03:43:ee:e9:58', 'links': [], 'enabled': True, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'ens4f0np0': {'type': 'physical', 'mac_address': 'f4:03:43:ef:10:60', 'links': [], 'enabled': True, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'ens4f1np1': {'type': 'physical', 'mac_address': 'f4:03:43:ef:10:68', 'links': [], 'enabled': True, 'parents': [], 'source': 'machine-resources', 'monitored': False}}
        Traceback (most recent call last):
          File "/snap/maas/19835/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 654, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/snap/maas/19835/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1475, in gotResult
            _inlineCallbacks(r, g, status)
          File "/snap/maas/19835/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/snap/maas/19835/usr/lib/python3/dist-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
        --- <exception caught here> ---
          File "/snap/maas/19835/lib/python3.8/site-packages/provisioningserver/utils/services.py", line 1091, in do_action
            yield self._updateInterfaces(interfaces)
          File "/snap/maas/19835/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/snap/maas/19835/usr/lib/python3/dist-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/snap/maas/19835/lib/python3.8/site-packages/provisioningserver/utils/services.py", line 1174, in _updateInterfaces
            yield self._run_refresh(
          File "/snap/maas/19835/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/snap/maas/19835/usr/lib/python3/dist-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/snap/maas/19835/lib/python3.8/site-packages/provisioningserver/utils/services.py", line 1201, in _run_refresh
            yield deferToThread(
          File "/snap/maas/19835/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
            result = inContext.theWork()
          File "/snap/maas/19835/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
            inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
          File "/snap/maas/19835/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/snap/maas/19835/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
            return func(*args,**kw)
          File "/snap/maas/19835/lib/python3.8/site-packages/provisioningserver/utils/twisted.py", line 202, in wrapper
            result = func(*args, **kwargs)
          File "/snap/maas/19835/lib/python3.8/site-packages/provisioningserver/refresh/__init__.py", line 63, in refresh
            failed_scripts = runscripts(
          File "/snap/maas/19835/lib/python3.8/site-packages/provisioningserver/refresh/__init__.py", line 172, in runscripts
            post_process_hook(
          File "/snap/maas/19835/lib/python3.8/site-packages/provisioningserver/utils/services.py", line 1224, in _annotate_commissioning
            lxd_data = json.loads(Path(stdout_path).read_bytes())
          File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
            return _default_decoder.decode(s)
          File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
            obj, end = self.raw_decode(s, idx=_w(s, 0).end())
          File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
            raise JSONDecodeError("Expecting value", s, err.value) from None
        json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

2022-09-16 12:12:59 twisted.internet.defer: [critical] Unhandled error in Deferred:
2022-09-16 12:12:59 twisted.internet.defer: [critical]
        Traceback (most recent call last):
        Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\xc0\xa8\x1e\x06', 98, 'Address already in use')

2022-09-16 12:12:59 twisted.internet.defer: [critical] Unhandled error in Deferred:
2022-09-16 12:12:59 twisted.internet.defer: [critical]
        Traceback (most recent call last):
        Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\n.\x12\x06', 98, 'Address already in use')
```

[Pedro Guimarães (pguimaraes)]

Raising this bug as field-critical as this issue is a show-stopper for us

[Przemyslaw Hausman (phausman)]

FYI, the problem is reproducible on a single, CIS-hardened focal VM.

[Przemyslaw Hausman (phausman)]

This is the CIS-hardening script I'm using: https://pastebin.canonical.com/p/d2nSpr3tXq/
Excuse the private pastebin but the script contains sensitive data and can't be shared publicly.

Steps to reproduce:

1. Spin up a new focal vm (based on http://releases.ubuntu.com/focal/ubuntu-20.04.5-live-server-amd64.iso)
2. Run CIS-hardening script
3. Install MAAS from snap (using maas-test-db is ok)
4. Watch /var/snap/maas/common/log/rackd.log for the error message

[Przemyslaw Hausman (phausman)]

For further testing I flushed all iptables rules and set default policy to ACCEPT on all 3 MAAS nodes. Then I reinstalled MAAS but the problem still persists.

[Przemyslaw Hausman (phausman)]

I have tracked down the rule that causes the problem. Run this before installing MAAS snap.

```
cp -v /usr/share/systemd/tmp.mount /etc/systemd/system/
sed -i 's/Options=.*/Options=mode=1777,strictatime,nosuid,nodev,noexec/gm' /etc/systemd/system/tmp.mount
systemctl daemon-reload
systemctl --now enable tmp.mount
```

[Przemyslaw Hausman (phausman)]

Apparently it is the `noexec` option on /tmp that is problematic. When I removed the `noexec` option, the problem was not showing up.

Removing this option is not a viable workaround though. CIS rule "1.1.5 Ensure noexec option set on /tmp partition (Automated)" requires us to have this option set.

[Christian Grabowski (cgrabowski)]

Able to confirm, the issue is the machine-resource script attempts to write the binary to a tmpfile in /tmp, which as previous comments mentioned are protected. This results in a permission denied error when the script then attempts to execute said binary.

[Adam Collard (adam-collard)]

Let's not use /tmp on MAAS controllers when executing machine-resources, instead use something in e.g. $MAAS_DATA - please check for other cases of using /tmp as a download + exec.

[Christian (cboitel)]

MAAS hardware synchronisation service suffer from /tmp with noexec.

I simply dropped a service override configuration to redefine TMPDIR environment variable.

I guess you could do alike to fix all of these issues globally : any mktemp syscall will then create files where you want (MAAS_DATA)