nova-api-metadata.service not getting request from compute node

[diarmuid]

nova-api-metadata.service not answering/responding to compute nodes
Actually the nova-api-metadata.service on the controller never gets the request from the compute node.

snap.microstack.nova-api-metadata.service

I can ping, and ssh to vm, but the user Key never gets inserted into the VM by the metadata process, so Im not able to log into the VM - on a compute node.

If the VM is created on the contorller node, I can access it. Also I see the nova-api-metadata.service executes a bunch of instructions against the VM.

ubuntu@racka37-controller:~$ microstack.openstack server create --flavor m1.medium --nic net-id=test --key-name microstack --image jammy experiment;
+-------------------------------------+----------------------------------------------+
| Field | Value |
+-------------------------------------+----------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| adminPass | M6wvHaTbZ9bC |
| config_drive | |
| created | 2022-09-23T16:51:07Z |
| flavor | m1.medium (3) |
| hostId | |
| id | 169bed0f-ab3d-4206-a402-ae473a18c76b |
| image | jammy (a64aa039-6e18-4b6d-a052-853a08f66d29) |
| key_name | microstack |
| name | experiment |
| progress | 0 |
| project_id | 147ee6614b2344a1932c5a41dddb031c |
| properties | |
| security_groups | name='default' |
| status | BUILD |
| updated | 2022-09-23T16:51:06Z |
| user_id | 35423533d7b54222aa104694a8fa535b |
| volumes_attached | |
+-------------------------------------+----------------------------------------------+

ubuntu@racka37-controller:~$ ALLOCATED_FIP=`microstack.openstack floating ip create -f value -c floating_ip_address external`
microstack.openstack server add floating ip my-microstack-server $ALLOCATED_FIP
No server with a name or ID of 'my-microstack-server' exists.
ubuntu@racka37-controller:~$ microstack.openstack server add floating ip experiment $ALLOCATED_FIP
ubuntu@racka37-controller:~$ microstack.openstack server delete e3286eec-e786-4f34-ab76-9c2142496379
ubuntu@racka37-controller:~$ microstack.openstack server list
+--------------------------------------+------------+--------+------------------------------------+-------+-----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------------+--------+------------------------------------+-------+-----------+
| 169bed0f-ab3d-4206-a402-ae473a18c76b | experiment | ACTIVE | test=192.168.222.146, 10.20.20.115 | jammy | m1.medium |
+--------------------------------------+------------+--------+------------------------------------+-------+-----------+
ubuntu@racka37-controller:~$ ping 10.20.20.115
PING 10.20.20.115 (10.20.20.115) 56(84) bytes of data.
64 bytes from 10.20.20.115: icmp_seq=1 ttl=63 time=3.99 ms
^C
--- 10.20.20.115 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 3.992/3.992/3.992/0.000 ms
ubuntu@racka37-controller:~$ ssh -i /home/ubuntu/snap/microstack/common/.ssh/id_microstack ubuntu@10.20.20.115
The authenticity of host '10.20.20.115 (10.20.20.115)' can't be established.
ED25519 key fingerprint is SHA256:DsYezNGyjcr6KQbFiZzz8wROpgY2ZFdla5LfRIJbPas.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.20.20.115' (ED25519) to the list of known hosts.
ubuntu@10.20.20.115: Permission denied (publickey).
ubuntu@racka37-controller:~$

ubuntu@racka37-controller:~$ microstack.openstack versions show
+-------------+---------------+---------+------------+---------------------------------+------------------+------------------+
| Region Name | Service Type | Version | Status | Endpoint | Min Microversion | Max Microversion |
+-------------+---------------+---------+------------+---------------------------------+------------------+------------------+
| microstack | identity | 3.14 | CURRENT | https:// :5000/v3/ | None | None |
| microstack | network | 2.0 | CURRENT | https:// :9696/v2.0/ | None | None |
| microstack | block-storage | 2.0 | DEPRECATED | https:// :8776/v2/ | None | None |
| microstack | block-storage | 3.0 | CURRENT | https:// :8776/v3/ | 3.0 | 3.60 |
| microstack | compute | 2.0 | SUPPORTED | https:// :8774/v2/ | None | None |
| microstack | compute | 2.1 | CURRENT | https:// :8774/v2.1/ | 2.1 | 2.87 |
| microstack | placement | 1.0 | CURRENT | https:// :8778/ | 1.0 | 1.36 |
| microstack | image | 2.0 | SUPPORTED | https:// :9292/v2/ | None | None |
| microstack | image | 2.1 | SUPPORTED | https:// :9292/v2/ | None | None |
| microstack | image | 2.2 | SUPPORTED | https:// :9292/v2/ | None | None |
| microstack | image | 2.3 | SUPPORTED | https:// :9292/v2/ | None | None |
| microstack | image | 2.4 | SUPPORTED | https:// :9292/v2/ | None | None |
| microstack | image | 2.5 | SUPPORTED | https:// :9292/v2/ | None | None |
| microstack | image | 2.6 | SUPPORTED | https:// :9292/v2/ | None | None |
| microstack | image | 2.7 | SUPPORTED | https:// :9292/v2/ | None | None |
| microstack | image | 2.9 | CURRENT | https:// :9292/v2/ | None | None |
+-------------+---------------+---------+------------+---------------------------------+------------------+------------------+
ubuntu@racka37-controller:~$