No functional tests covering the local multi-node scenario

[Nobuto Murata]

I think this is a known gap in the development team, but as of today there is no functional test covering the local multi-node scenario described in the tutorial:
https://microstack.run/docs/multi-node

https://github.com/canonical/snap-openstack/blob/f24943b0d6c26929967a21d1dbbb53ff5c75d7f9/.github/workflows/build-snap.yml#L51
> sg snap_daemon "openstack.sunbeam cluster bootstrap --manifest /snap/openstack/current/etc/manifests/edge.yml --accept-defaults --topology single --database single"

2024-05-18T01:33:56.7480160Z Node has been bootstrapped with roles: control, compute
2024-05-18T01:34:00.4966471Z ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
2024-05-18T01:34:00.4968490Z ┃ Node ┃ Status ┃ Control ┃ Compute ┃ Storage ┃
2024-05-18T01:34:00.4970542Z ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
2024-05-18T01:34:00.4972821Z │ large-422-e271738c9fe1e6272f349999.lxd │ up │ x │ x │ │
2024-05-18T01:34:00.4974758Z └────────────────────────────────────────┴────────┴─────────┴─────────┴─────────┘

By having the func test, the following issues could have been discovered earlier.
- https://bugs.launchpad.net/snap-openstack/+bug/2065470
- https://bugs.launchpad.net/snap-openstack/+bug/2065700
- https://bugs.launchpad.net/snap-openstack/+bug/2065866
- https://bugs.launchpad.net/snap-openstack/+bug/2065855

[Nobuto Murata (nobuto)]

For the record, here is how I'm testing the multi-node scenario locally. And the whole process takes 115 minutes.

for i in {1..3}; do
    uvt-kvm wait "sunbeam-${i}.localdomain"

    uvt-kvm ssh "sunbeam-${i}.localdomain" -- -t sudo snap install openstack --channel 2024.1/edge
    uvt-kvm ssh "sunbeam-${i}.localdomain" -- 'sunbeam prepare-node-script | bash -x'

    # LP: #2065911
    # TODO: make it permanent across reboots
    uvt-kvm ssh "sunbeam-${i}.localdomain" -- sudo ip link set enp9s0 up
done

uvt-kvm ssh sunbeam-1.localdomain -- tee deployment_manifest.yaml < manifest.yaml
uvt-kvm ssh sunbeam-1.localdomain -- 'tail -n+2 /snap/openstack/current/etc/manifests/edge.yml >> deployment_manifest.yaml'

uvt-kvm ssh sunbeam-1.localdomain -- -t \
    time sunbeam cluster bootstrap --manifest deployment_manifest.yaml \
        --role control --role compute --role storage

## -> real 21m13.010s

# LP: #2065490
uvt-kvm ssh sunbeam-1.localdomain -- 'juju model-default --cloud sunbeam-microk8s logging-config="<root>=INFO;unit=DEBUG"'
uvt-kvm ssh sunbeam-1.localdomain -- 'juju model-config -m openstack logging-config="<root>=INFO;unit=DEBUG"'

uvt-kvm ssh sunbeam-2.localdomain -- -t \
    time sunbeam cluster join --role control --role compute --role storage \
        --token "$(uvt-kvm ssh sunbeam-1.localdomain -- sunbeam cluster add --name sunbeam-2.localdomain -f value)"

## -> real 25m3.417s

uvt-kvm ssh sunbeam-3.localdomain -- -t \
    time sunbeam cluster join --role control --role compute --role storage \
        --token "$(uvt-kvm ssh sunbeam-1.localdomain -- sunbeam cluster add --name sunbeam-3.localdomain -f value)"

## -> real 27m28.257s

uvt-kvm ssh sunbeam-1.localdomain -- -t \
    time sunbeam cluster resize

## -> real 32m55.238s

uvt-kvm ssh sunbeam-1.localdomain -- -t \
    time sunbeam configure --openrc demo-openrc --manifest deployment_manifest.yaml

## -> real 3m30.742s

uvt-kvm ssh sunbeam-1.localdomain -- -t \
    'time sunbeam openrc > admin-openrc'

## -> real 0m11.547s

uvt-kvm ssh sunbeam-1.localdomain -- -t \
    time sunbeam launch ubuntu --name test

## -> real 1m13.650s