Failed to create port on network <UUID>, because fixed_ips included invalid subnet <UUID>

[Felipe Reyes]

[Impact]

When creating a new "cluster"that overrides the fixed network defined in the cluster template, but not the subnet.

It would be expected that the cluster create request fails with a 400 error since the client is submitting an invalid request.

[Environment]

Focal Ussuri

[Test Case]

1. Create a new cluster template WITHOUT a fixed network/subnet set.
openstack coe cluster template create k8s-cluster-template \
    --image fedora-coreos-32 \
    --keypair testkey \
    --external-network ext_net \
    --flavor m1.small \
    --network-driver flannel \
    --coe kubernetes \
    --fixed-network admin_net \
    --fixed-subnet admin_subnet

2. Create a new cluster using the template previously created and select an existing network

openstack coe cluster create \
    --cluster-template k8s-cluster-template \
    --timeout 120 \
    --fixed-network private \
    k8scluster

Expected result

The cluster gets created

Actual result:

The cluster creation fails with the following error:

$ openstack coe cluster show k8scluster -f json -c faults | jq -r '.faults'
{
  "default-master": "Resource CREATE failed: BadRequest: resources.kube_masters.resources[0].resources.kube_master_eth0: Invalid input for operation: Failed to create port on network 525df7a4-1aeb-4eae-a37f-432a809a8161, because fixed_ips included invalid subnet 30e1b4ed-811f-4226-a19d-0a56cc72fc10.\nNeutron server returns request_ids: ['req-7a55a40a-3aa3-4a67-8ecf-b2e47ae16a84']",
  "default-worker": "Resource CREATE failed: BadRequest: resources.kube_masters.resources[0].resources.kube_master_eth0: Invalid input for operation: Failed to create port on network 525df7a4-1aeb-4eae-a37f-432a809a8161, because fixed_ips included invalid subnet 30e1b4ed-811f-4226-a19d-0a56cc72fc10.\nNeutron server returns request_ids: ['req-7a55a40a-3aa3-4a67-8ecf-b2e47ae16a84']"
}

[Felipe Reyes (freyes)]

the issue can be reproduced using master (with devstack)

$ openstack coe cluster template create k8s-cluster-template --image fedora-coreos-35.20220116.3.0-openstack.x86_64 --keypair testkey --external-network public --flavor m1.small --network-driver flannel --coe kubernetes
Request to create cluster template k8s-cluster-template accepted
+-----------------------+------------------------------------------------+
| Field | Value |
+-----------------------+------------------------------------------------+
| insecure_registry | - |
| labels | {} |
| updated_at | - |
| floating_ip_enabled | True |
| fixed_subnet | - |
| master_flavor_id | - |
| uuid | 9c7edc11-a710-4f24-8850-f9911e7ff0d5 |
| no_proxy | - |
| https_proxy | - |
| tls_disabled | False |
| keypair_id | testkey |
| public | False |
| http_proxy | - |
| docker_volume_size | - |
| server_type | vm |
| external_network_id | public |
| cluster_distro | fedora-coreos |
| image_id | fedora-coreos-35.20220116.3.0-openstack.x86_64 |
| volume_driver | - |
| registry_enabled | False |
| docker_storage_driver | overlay2 |
| apiserver_port | - |
| name | k8s-cluster-template |
| created_at | 2023-10-03T19:45:21.945595+00:00 |
| network_driver | flannel |
| fixed_network | - |
| coe | kubernetes |
| flavor_id | m1.small |
| master_lb_enabled | False |
| dns_nameserver | 8.8.8.8 |
| hidden | False |
| tags | - |
+-----------------------+------------------------------------------------+
$ openstack coe cluster create \
    --cluster-template k8s-cluster-template \ ...

[Felipe Reyes (freyes)]

The approach I'm taking to solve this problem is to implement the validate_fixed_network and validate_fixed_subnet - https://opendev.org/openstack/magnum/src/commit/a3276fd53b3a018522e3159137664d9808603514/magnum/api/attr_validator.py#L99

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/magnum/+/897362

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/magnum/+/897506

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/magnum/+/897507

[Felipe Reyes (freyes)]

There is an issue with magnum-ui preventing users from selecting a subnet when overriding the network, see https://bugs.launchpad.net/magnum-ui/+bug/2038663

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/magnum-ui/+/898007

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/magnum/+/897362
Committed: https://opendev.org/openstack/magnum/commit/753baadbb8b5b4c3032d4618166b1c899a50fb07
Submitter: "Zuul (22348)"
Branch: master

commit 753baadbb8b5b4c3032d4618166b1c899a50fb07
Author: Felipe Reyes <email address hidden>
Date: Thu Oct 5 18:39:39 2023 -0300

    Add validator for fixed_network.

    The validate_fixed_network() validator looks for an available network
    matching the id or name, if no networks match a FixedNetworkNotFound
    exception is raised, if more than one networks match then the Conflict
    exception is raised.

    Partial-Bug: #2038109
    Change-Id: I7fb0122889feb4f8fa039be5752e8ac3fbc23b94

[Jake Yip (waipengyip)]

Hi Felipe,

Thanks for finding this. Looks like fixed_network and fixed_subnet must both be specified, else things break apart at the heat driver layer.

- fixed-network without fixed-subnet = heat can't find subnet with error "Multiple subnet matches found for name '', use an ID to be more specific."
- fixed-subnet without fixed-network = heat template doesn't use the subnet due to the detection at [1]

I think it is fair that there are some validation to make sure network / subnet exist. But I'm not sure if we should validate that both of them exist together in the api/attr_validation.py layer. Future driver implementation may only need one and not the other. If Heat needs both, Heat should validate it.

Can we move forward by simply validating subnet exists like how you did with network?

[1] https://opendev.org/openstack/magnum/src/branch/stable/2023.2/magnum/drivers/heat/template_def.py#L591

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/magnum-ui/+/898007
Committed: https://opendev.org/openstack/magnum-ui/commit/6f6c3db282fe2f0e08ad69c557eb153858b0164a
Submitter: "Zuul (22348)"
Branch: master

commit 6f6c3db282fe2f0e08ad69c557eb153858b0164a
Author: Felipe Reyes <email address hidden>
Date: Tue Oct 10 18:33:27 2023 -0300

    Add dropdown list to select fixed subnet

    When creating a cluster and the user decides to reuse an existing
    network, it must also choose a subnet, otherwise the resulting
    configuration may be invalid when trying to allocate a port.

    Closes-Bug: #2038663
    Related-Bug: #2038109
    Change-Id: I08943f10418385a39eecc7b4117d162854d2d010

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/magnum-ui/+/905083

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/magnum/+/897506
Committed: https://opendev.org/openstack/magnum/commit/a8bce0bfee81218cd1c0ddcf3e2b86b96659933e
Submitter: "Zuul (22348)"
Branch: master

commit a8bce0bfee81218cd1c0ddcf3e2b86b96659933e
Author: Felipe Reyes <email address hidden>
Date: Thu Oct 5 18:45:55 2023 -0300

    Add validator for fixed_subnet

    Validate the existance of the subnet referenced by fixed_subnet. It's
    not checked if the subnet is associated to the fixed_network.

    Closes-Bug: #2038109
    Change-Id: Ia75f0ae525b768ad5b965d22b522cca6f80dcab2

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/magnum 18.0.0.0rc1 release candidate.