Pod must be on a known host if interfaces are specified

[Marian Gasparovic]

maas_2.6.1~rc1-7830-gee83011f2-ubuntu1~18.04.1
getting this error now

2019-10-11-06:35:41 root DEBUG maas root pod compose 1 hostname=vault-1 cores=2 memory=4096 storage=20.0 zone=1 interfaces=eth0:space=oam-space;eth1:space=internal-space,type=bridge
2019-10-11-06:35:42 root ERROR Command failed: pod compose 1 hostname=vault-1 cores=2 memory=4096 storage=20.0 zone=1 interfaces=''eth0:space=oam-space;eth1:space=internal-space,type=bridge''
2019-10-11-06:35:42 root ERROR b'Pod must be on a known host if interfaces are specified.'
Traceback (most recent call last):
  File "/usr/local/bin/fce", line 11, in <module>
    load_entry_point('foundationcloudengine', 'console_scripts', 'fce')()
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/main.py", line 141, in entry_point
    sys.exit(main(sys.argv[1:]))
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/main.py", line 132, in main
    opts.func(opts)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/build.py", line 73, in build_main
    args.steps)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/build.py", line 47, in build_and_validate_if_needed
    layer.build_outer(only_steps)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/layers/baselayer.py", line 119, in build_outer
    self.build(only_steps=only_steps)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/layers/maaslayer.py", line 2662, in build
    super(MaasLayer, self).run_steps(only_steps)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/layers/steppedbaselayer.py", line 51, in run_steps
    step.build()
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/layers/maaslayer.py", line 1368, in build
    zone['id'],
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/maas_cli.py", line 515, in add_pod_vm
    return cmd(maas_profile, command)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/maas_cli.py", line 117, in cmd
    raise error
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/maas_cli.py", line 112, in cmd
    output = raw_cmd(maas_profile, split_command)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/maas_cli.py", line 106, in raw_cmd
    return subprocess.check_output(maas_cmd)
  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['maas', 'root', 'pod', 'compose', '1', 'hostname=vault-1', 'cores=2', 'memory=4096', 'storage=20.0', 'zone=1', 'interfaces=eth0:space=oam-space;eth1:space=internal-space,type=bridge']' returned non-zero exit status 2.

[Marian Gasparovic (marosg)]

subsribed also field-high

[Blake Rouse (blake-rouse)]

Is the Pod created on a machine that was deployed with "Install as KVM host"? If not then you cannot use interface constraints.

[Marian Gasparovic (marosg)]

Blake, when I look at a history of our test runs I see we get this working sometimes and failing sometimes, on the same maas revision

2019-10-13-13:09:30 foundationcloudengine.layers.maaslayer INFO Creating vault-1 in leafeon
2019-10-13-13:09:30 root DEBUG maas root pod compose 1 hostname=vault-1 cores=2 memory=4096 storage=20.0 zone=1 interfaces=eth0:space=oam-space;eth1:space=internal-space,type=bridge
2019-10-13-13:09:37 root DEBUG maas root tag read vault
2019-10-13-13:09:38 root DEBUG maas root tags create name=vault
2019-10-13-13:09:39 root DEBUG maas root tag update-nodes vault add=fepak3

[Jason Hobbs (jason-hobbs)]

Some more background - we've been running this way on maas 2.6.0 for sometime now and never hit this issue. FCB's design depends on using pods to run on the maas rack/region controllers, and they need to be on multiple interfaces.

We've always done that before and it's worked; we don't expect change behavior like this in a point release.

[Blake Rouse (blake-rouse)]

Nothing has changed in behavior, the behavior has alway been this way. It is either you install the Pod using "Install as KVM host" or if the machine was deployed by MAAS then MAAS should be able to determine that the machine is the host of the Pod.

I assume you are in the later case?

[Jason Hobbs (jason-hobbs)]

The behavior has certainly changed in 2.6.1, we've run hundreds of test cases prior to that and haven't seen this behavior before.

The machine wasn't deployed by MAAS - it is the machine hosting MAAS, in fact. We add it to MAAS as a pod host.

I'm not sure what this error even means, what is it trying to tell us?

[Jason Hobbs (jason-hobbs)]

Some more background:

We're hosting pods on infra nodes - host maas rack/region controller.
After maas is deployed, we add the three infra nodes as pod hosts
The maas rack controller has interfaces on two networks - OAM for PXE, internal to talk to openstack services.
We need the pod kvm to have interfaces on both networks, so we pass both in during compose. We pass in two nic constraints because we want two interfaces.
Due to bug 1830690, the interface is unconfigured. we go and link it to a subnet via the api.

[Newell Jensen (newell-jensen)]

As this is a randomly failing and has to due with interface constraints I have reverted commit dbbc280d218fb67cbaf6e1acb30b1549e7c110a5n and have added a 2.6.1 package to ppa:maas-maintainers/experimental3

This will get you on your way while I continue to look into what is causing this regression.

[Jason Hobbs (jason-hobbs)]

Update: we hit the issue on the new package from the experimental3 ppa. Newell has access to the repro system and is investigating.

[Alexander Balderson (asbalderson)]

Update:
Using the New New package from experimental3 ppa i had 3 successful builds in a row.

[Newell Jensen (newell-jensen)]

Update:

soultions-qa has not seen any regressions with 2.6.0 but has seen regressions with 2.6.1 as well 2.6.1 with commit dbbc280d218fb67cbaf6e1acb30b1549e7c110a5n reverted. 2.6.0 did not show any issues but the other packaages did. Currently testing 2.6.1-alpha1 (aka 2.6.0-rc3).

[Adam Collard (adam-collard)]

After many more runs in testing, we cannot reproduce the issue. If it comes back please alert us immediately, and reopen the bug.

[Marian Gasparovic (marosg)]

We just hit it again last night with maas_2.6.1-7832-g17912cdc9-0ubuntu1~18.04.1

2019-10-31-08:28:28 root DEBUG maas root pod compose 1 hostname=vault-1 cores=2 memory=4096 storage=20.0 zone=2 interfaces=eth0:space=oam-space;eth1:space=internal-space,type=bridge
2019-10-31-08:28:29 root ERROR Command failed: pod compose 1 hostname=vault-1 cores=2 memory=4096 storage=20.0 zone=2 interfaces=''eth0:space=oam-space;eth1:space=internal-space,type=bridge''
2019-10-31-08:28:29 root ERROR b'Pod must be on a known host if interfaces are specified.'

[Marian Gasparovic (marosg)]

and it passed a day before with 2.6.1~rc1-7830-gee83011f2-ubuntu1~18.04.1

[Newell Jensen (newell-jensen)]

After being able to take a closer look at a failing system the issue is that the rack controllers are not able to update their interfaces. The underlying issue seems to be that the networking monitoring services has a lock file to ensure that only one process updates the networking information. If the processes gets killed, the lock file stays, pointing to the PID the killed regiond process had.

Now what normally happens is that another process tries to acquire the lock, sees that the lock points to a killed PID , and recreates the lock.

This normally works, but what can happen is that the killed PID gets recycled, so that the lock now points to a PID which the maas user isn't allowed to kill. Now a PermissionError is raised, that the lock file implementation doesn't handle this case, and the networking monitoring service can never start.

Currently working on a fix for this.

[Michael Skalka (mskalka)]

We have encountered this error again in testing using maas_2.6.2-7841-ga10625be3-0ubuntu1~18.04.1

MAAS logs attached.

[Michael Skalka (mskalka)]

Foundation log from the failed run above.

[Alexander Balderson (asbalderson)]

I'm reopening this; as we're seeing it on maas_2.7.0~rc1-8204-g.d93c8433c-0ubuntu1~18.04.1

[Michael Skalka (mskalka)]

Full artifacts here:
https://oil-jenkins.canonical.com/artifacts/bbefe487-c887-4f85-927b-fd05aef88038/index.html

Attaching the maas-specific logs.

[Björn Tillenius (bjornt)]

I think we've found the real issue now. In Controller.update_interfaces(), we loop over the detected interfaces, first the physical interfaces, and then the vlans, bonds, and bridges. In the loop we get the interface details via mac address.

However, only physical interfaces have extra details. If a bridge, bond or vlan has the same mac address as a physical one, it will get its extra details as well, including the name.

The fix is to only update extra information if it's a physical interface.

The code in question has changed from 2.6.2 and 2.7rc1, but the underlying issue exists in both. So both 2.6 and 2.7 need to be fixed.

I was able to reproduce this with a physical machine (a container won't do) where a single rack and region was installed. Stop maas-rackd and restart maas-regiond, and you should see the issue.

The reason it didn't happen on every test run is that the code works fine the first time it runs, when all the interfaces are created. Then either rackd or regiond will have the responsibility to keep the network information up-to-date, and if the service that doesn't have the responsibility is restarted, the issue won't show, since it won't try to update the interfaces.

[John George (jog)]

Solutions QA is continuing to hit this bug. This link has all the test runs that have experienced the failure: https://solutions.qa.canonical.com/#/qa/bug/1847794

Individual test runs are linked from this page. Once on the test run page the artifacts and logs are linked at the bottom.

Here's a direct link to one of the recent failures:
https://solutions.qa.canonical.com/#/qa/testRun/e04fb3be-07f4-4e65-ad3d-eab2a1d34b17

[Adam Collard (adam-collard)]

See also https://bugs.launchpad.net/maas/+bug/1859210

[Michael Skalka (mskalka)]

Tagging this as a release blocker for 2.8rc5 as we have not been able to get past pod composition for the full test run due to this issue.

[Michael Skalka (mskalka)]

Subbing field-critical, we have hit this 11 times in the last ~24 hours. https://solutions.qa.canonical.com/#/qa/bug/1847794

[Björn Tillenius (bjornt)]

Ok, we found the issue. The problem happens when the maas deb packages are purged and then reinstalled. That might leave maas lock files in /run/lock/ that are now owned by a numeric user.

If maas is reinstalled directly, then things should work since maas should get the same uid as before and things will work.

But if you install some other deb package that adds a user, and then reinstall maas, the maas uid will now be different than before, and it won't have permissions to remove the stale lock files.

So I would say that the original bug that this was about was actually fixed already.

This is a separate bug that probably started to show up when you changed your systems in some way (i.e. installing an extra package), having the same symptoms as the original bug.

That explains why you started to see it recently only, after not seeing it for a long while.

[Alberto Donato (ack)]

Filed LP:#1883735 to track the issue mentioned above.

[Michael Skalka (mskalka)]

Suggested workaround confirmed. Removing tags and subscriptions. We'll hold the fix until the package bug is resolved.

[Adam Collard (adam-collard)]

Bug 1883735 was the root cause of this

[Camille Rodriguez (camille.rodriguez)]

Hi, I'm currently hitting this issue on a new install of maas 2.8 (2.8.2-8577-g.a3e674063).

2021-02-26-22:40:41 root ERROR [localhost] Command failed: maas root pod compose 2 hostname=vault-2 cores=2 memory=4096 storage=40.0 zone=1 'interfaces=eth0:space=oam-space,mode=auto;eth1:space=internal-space
,type=bridge,mode=auto'
2021-02-26-22:40:41 root ERROR [localhost] STDOUT follows:
Pod must be on a known host if interfaces are specified

I don't think the bug 1883735 is the root cause in my case since this is a fresh maas install on a fresh ubuntu install.

[Marian Gasparovic (marosg)]

I hit it today when testing 3.3.0~rc2-13144-g.b2d51cb8c

2023-01-17-08:35:24 root ERROR [localhost] Command failed: maas root vm-host compose 1 hostname=vault-1 cores=2 memory=4096 storage=40.0 zone=2 'interfaces=eth0:space=oam-space;eth1:space=internal-space'
2023-01-17-08:35:24 root ERROR [localhost] STDOUT follows:
Pod must be on a known host if interfaces are specified.

[Marian Gasparovic (marosg)]

And hit it again, here are the logs

https://oil-jenkins.canonical.com/artifacts/1d7331da-7378-4720-8663-a3b392173c42/generated/generated/maas/logs-2023-01-18-07.56.03.tgz

[Bartłomiej Poniecki-Klotz (barteus)]

We hit the same issue on the customer environment with the same VM (vault-1). Retrying the step did not help, the solution was to clean the whole layer in FCE and redeploy again.

OS
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"

Maas version: 3.1.1-10918-g.9cbd96fd2

[Bartłomiej Poniecki-Klotz (barteus)]

I'm hitting this issue again while redeploying the MAAS layer. Cleanup of the layer and redeployment do not help. This always fails on the same VM.

OS: Ubuntu 22.04
MAAS: 3.1.2

Log:
2023-12-06-08:04:16 root ERROR [localhost] Command failed: maas root vm-host compose 3 hostname=vault-2 cores=2 memory=4096 storage=40.0 zone=3 'interfaces=eth0:space=oam-space;eth1:space=internal-space'
2023-12-06-08:04:16 root ERROR 1[localhost] STDOUT follows:
Pod must be on a known host if interfaces are specified.
2023-12-06-08:04:16 root ERROR 2[localhost] STDERR follows:
b''
Traceback (most recent call last):
  File "/usr/local/bin/fce", line 33, in <module>
    sys.exit(load_entry_point('foundationcloudengine', 'console_scripts', 'fce')())
  File "/home/ubuntu/fce/foundationcloudengine/foundationcloudengine/main.py", line 217, in entry_point
    sys.exit(main(sys.argv[1:]))
  File "/home/ubuntu/fce/foundationcloudengine/foundationcloudengine/main.py", line 208, in main
    opts.func(opts)
  File "/home/ubuntu/fce/foundationcloudengine/foundationcloudengine/command.py", line 91, in run
    self.run_per_layer(layer, args)
  File "/home/ubuntu/fce/foundationcloudengine/foundationcloudengine/commands/build.py", line 46, in run_per_layer
    build_and_validate_if_needed(
  File "/home/ubuntu/fce/foundationcloudengine/foundationcloudengine/commands/build.py", line 98, in build_and_validate_if_needed
    layer.build_outer(only_steps)
  File "/home/ubuntu/fce/foundationcloudengine/foundationcloudengine/layers/baselayer.py", line 160, in build_outer
    self.build(only_steps=only_steps)
  File "/home/ubuntu/fce/foundationcloudengine/foundationcloudengine/layers/maaslayer.py", line 2057, in build
    super().run_steps(only_steps)
  File "/home/ubuntu/fce/foundationcloudengine/foundationcloudengine/layers/steppedbaselayer.py", line 110, in run_steps
    step.build()
  File "/home/ubuntu/fce/foundationcloudengine/foundationcloudengine/layers/configuremaas.py", line 950, in build
    created_vm = add_vm_host_vm(
  File "/home/ubuntu/fce/foundationcloudengine/foundationcloudengine/maas_cli.py", line 707, in add_vm_host_vm
    return local_parse_json(command)
  File "/home/ubuntu/fce/foundationcloudengine/foundationcloudengine/remotehelpers.py", line 300, in local_parse_json
    return json.loads(local(cmd, **kwargs))
  File "/home/ubuntu/fce/foundationcloudengine/foundationcloudengine/remotehelpers.py", line 278, in local
    return run_cmd(cmd, target_machine, **kwargs)
  File "/home/ubuntu/fce/foundationcloudengine/foundationcloudengine/remotehelpers.py", line 209, in run_cmd
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['maas', 'root', 'vm-host', 'compose', '3', 'hostname=vault-2', 'cores=2', 'memory=4096', 'storage=40.0', 'zone=3', 'interfaces=eth0:space=oam-space;eth1:space=internal-space']' returned non-zero exit status 2.

When I run command manually, I get:
$ maas root vm-host compose 3 hostname=vault-2 cores=2 memory=4096 storage=40.0 zone=3 'interfaces=eth0:space=oam-space;eth1:space=internal-space'
Pod must be on a known host if interfaces are specified.

[Ancheng Liu (anchengliu)]

I hit it today when deploying pod on infra host(MaaS running on it) via fce command
$ fce --debug build --layer maas --steps \
maas:setup_vm_host,maas:setup_infra_kvm,maas:deploy_kvm_nodes,maas:compose_vms

OS: ubuntu22.04.3 LTS
MaaS: 3.3.4

2023-12-22-04:19:37 root ERROR [localhost] Command failed: maas root vm-host compose 2 hostname=vault-2 cores=4 memory=8192 storage=80.0 zone=3 'interfaces=eth0:space=oam-space;eth1:space=internal-space'
2023-12-22-04:19:37 root ERROR 1[localhost] STDOUT follows:
Pod must be on a known host if interfaces are specified.
2023-12-22-04:19:37 root ERROR 2[localhost] STDERR follows:
b''
Traceback (most recent call last):
  File "/usr/local/bin/fce", line 33, in <module>
    sys.exit(load_entry_point('foundationcloudengine', 'console_scripts', 'fce')())
  File "/home/ubuntu/cpe-foundation/foundationcloudengine/foundationcloudengine/main.py", line 217, in entry_point
    sys.exit(main(sys.argv[1:]))
  File "/home/ubuntu/cpe-foundation/foundationcloudengine/foundationcloudengine/main.py", line 208, in main
    opts.func(opts)
  File "/home/ubuntu/cpe-foundation/foundationcloudengine/foundationcloudengine/command.py", line 91, in run
    self.run_per_layer(layer, args)
  File "/home/ubuntu/cpe-foundation/foundationcloudengine/foundationcloudengine/commands/build.py", line 46, in run_per_layer
    build_and_validate_if_needed(
  File "/home/ubuntu/cpe-foundation/foundationcloudengine/foundationcloudengine/commands/build.py", line 98, in build_and_validate_if_needed
    layer.build_outer(only_steps)
  File "/home/ubuntu/cpe-foundation/foundationcloudengine/foundationcloudengine/layers/baselayer.py", line 160, in build_outer
    self.build(only_steps=only_steps)
  File "/home/ubuntu/cpe-foundation/foundationcloudengine/foundationcloudengine/layers/maaslayer.py", line 2001, in build
    super().run_steps(only_steps)
  File "/home/ubuntu/cpe-foundation/foundationcloudengine/foundationcloudengine/layers/steppedbaselayer.py", line 110, in run_steps
    step.build()
  File "/home/ubuntu/cpe-foundation/foundationcloudengine/foundationcloudengine/layers/configuremaas.py", line 890, in build
    created_vm = add_vm_host_vm(
  File "/home/ubuntu/cpe-foundation/foundationcloudengine/foundationcloudengine/maas_cli.py", line 693, in add_vm_host_vm
    return local_parse_json(command)
  File "/home/ubuntu/cpe-foundation/foundationcloudengine/foundationcloudengine/remotehelpers.py", line 293, in local_parse_json
    return json.loads(local(cmd, **kwargs))
  File "/home/ubuntu/cpe-foundation/foundationcloudengine/foundationcloudengine/remotehelpers.py", line 271, in local
    return run_cmd(cmd, target_machine, **kwargs)
  File "/home/ubuntu/cpe-foundation/foundationcloudengine/foundationcloudengine/remotehelpers.py", line 202, in run_cmd
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['maas', 'root', 'vm-host', 'compose', '2', 'hostname=vault-2', 'cores=4', 'memory=8192', 'storage=80.0', 'zone=3', 'interfaces=eth0:space=oam-space;eth1:space=internal-space']' returned non-zero exit status 2.