Compute node instances unable to reach metadata endpoint

[Andrew Miller]

Getting the following error in a multinode setup in instances running on compute nodes. Instances scheduled on the control node are able to connect to http://169.254.169.254/2009-04-04/instance-id.

failed to read iid from metadata. tried 20

The above error results in nodes being unable to get SSH key which makes juju impossible to install. Also using these instances is somewhat harder because the SSH key does not get injected into the system. Instances are able to communicate to each other though so the network seems to be working just fine.

Below are logs from an instance running on a compute node. Control node runs on Pop!_OS 20.04 LTS. Compute nodes run on focal and hirsute (pop os).

info: initramfs: up at 0.56
modprobe: module virtio_pci not found in modules.dep
modprobe: module virtio_blk not found in modules.dep
modprobe: module virtio_net not found in modules.dep
modprobe: module vfat not found in modules.dep
modprobe: module nls_cp437 not found in modules.dep
info: copying initramfs to /dev/vda1
info: initramfs loading root from /dev/vda1
info: /etc/init.d/rc.sysinit: up at 0.70
info: container: none
Starting logging: OK
modprobe: module virtio_pci not found in modules.dep
modprobe: module virtio_blk not found in modules.dep
modprobe: module virtio_net not found in modules.dep
modprobe: module vfat not found in modules.dep
modprobe: module nls_cp437 not found in modules.dep
WARN: /etc/rc3.d/S10-load-modules failed
Initializing random number generator... [ 0.849150] random: dd urandom read with 17 bits of entropy available
done.
Starting acpid: OK
Starting network...
udhcpc (v1.23.2) started
Sending discover...
Sending select for 192.168.222.253...
Lease of 192.168.222.253 obtained, lease time 43200
route: SIOCADDRT: File exists
WARN: failed: route add -net "0.0.0.0/0" gw "192.168.222.1"
checking http://169.254.169.254/2009-04-04/instance-id
failed 1/20: up 0.78. request failed
failed 2/20: up 12.78. request failed
failed 3/20: up 24.79. request failed
failed 4/20: up 36.80. request failed
failed 5/20: up 48.81. request failed
failed 6/20: up 60.82. request failed
failed 7/20: up 72.82. request failed
failed 8/20: up 84.83. request failed
failed 9/20: up 96.84. request failed
failed 10/20: up 108.85. request failed
failed 11/20: up 120.86. request failed
failed 12/20: up 132.86. request failed
failed 13/20: up 144.87. request failed
failed 14/20: up 156.88. request failed
failed 15/20: up 168.89. request failed
failed 16/20: up 180.90. request failed
failed 17/20: up 192.91. request failed
failed 18/20: up 204.91. request failed
failed 19/20: up 216.92. request failed
[ 230.442452] random: nonblocking pool is initialized
failed 20/20: up 228.93. request failed
failed to read iid from metadata. tried 20
failed to get instance-id of datasource
Top of dropbear init script
Starting dropbear sshd: failed to get instance-id of datasource
OK
GROWROOT: CHANGED: partition=1 start=18432 old: size=71647 end=90079 new: size=2078687,end=2097119
=== system information ===
Platform: OpenStack Foundation OpenStack Nova
Container: none
Arch: x86_64
CPU(s): 1 @ 3593.250 MHz
Cores/Sockets/Threads: 1/1/1
Virt-type: AMD-V
RAM Size: 488MB
Disks:
NAME MAJ:MIN SIZE LABEL MOUNTPOINT
vda 253:0 1073741824
vda1 253:1 1064287744 cirros-rootfs /
vda15 253:15 8388608
=== sshd host keys ===
-----BEGIN SSH HOST KEY KEYS-----
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCP3WAHVN3Soem1LaQlGA3gvLwIn1FeRgYXejC31bt7nCx6sTQgTQJHvdLuufHyY3wySLWBBUJmSXbYDwLuJ0NI9ZUNZl66EvC97UsVi2rfOb8nftoFq0B7Xbf0UIdxTAKvLI4zeeFjhyXZj+Qqp+xWZBbq6YzbHVLrueZ86CxzCcZZptQjYRIqjeiV8eYWhvx/4mQaUi10XSD4Fgp/YZug+sn9QJfihLCAX1L5MSSPQ9mE5wLJdpaTyrEbGS/M5UTGCEe7NoLAB/2d0MkeMqPgQzDEzseNj5PaDVYdMxIRlrvYSfyD5JpJ3yOVr2LlS2xKRO1dTIrzYxY0wF+01FfF root@cirros
ssh-dss AAAAB3NzaC1kc3MAAACBANNnjAErwOm1Op4rcWAjoac9l+EML3u8F34VHD5V1EsbXWambe4TXNErCnHFYdV2oF57QgmJfXjwRGy5JwZI4SX84yQd5dL5fEY73u0Vgaioj2DR2bNuKbl2d9I5VQpISnvFHN2gMpkiipYVzp5a0rqprnV+vQJ8x/DFyNqkDUtRAAAAFQCdW/KxTws+6OJC/Z1PFSTCkYV7EwAAAIAeISRANIf6qRMh2mLZwtCAnt936XLC4QGkwPpzvaZGA8VuFj71C6S5DqmCeG/cqridtCoppFstVOOcRfRmUdA/Zk41Z9j6bA9PIbE7J2umBjw32+rA9AxAgw+G0qNFKqIf2vxkYNW9hXhv4EI+MNVXxz55lOM3TFcDSQXA/qoPigAAAIBbdjQVXZ9kgwv6W8O1Ro6jnoIblIQIhqoAcUosQq9ftckzFYus0vprPURJNbN1f6XWY06cRPHD4GJOYVbww0gUjATZGsUebasAigHnsUCnG7XzNJ1Zn4TMr/dryas5FnRd/TW0GFl3fvjkg2d+G+2Qtp4w9fxcAYXhKW3nuy14Bw== root@cirros
-----END SSH HOST KEY KEYS-----
=== network info ===
if-info: lo,up,127.0.0.1,8,,
if-info: eth0,up,192.168.222.253,24,fe80::f816:3eff:feeb:f0bd/64,
ip-route:default via 192.168.222.1 dev eth0
ip-route:169.254.169.254 via 192.168.222.2 dev eth0
ip-route:192.168.222.0/24 dev eth0 src 192.168.222.253
ip-route6:fe80::/64 dev eth0 metric 256
ip-route6:unreachable default dev lo metric -1 error -101
ip-route6:ff00::/8 dev eth0 metric 256
ip-route6:unreachable default dev lo metric -1 error -101
=== datasource: None None ===
=== cirros: current=0.4.0 uptime=241.67 ===
  ____ ____ ____
 / __/ __ ____ ____ / __ \/ __/
/ /__ / // __// __// /_/ /\ \
\___//_//_/ /_/ \____/___/
   http://cirros-cloud.net

/dev/root resized successfully [took 0.13s]

login as 'cirros' user. default password: 'gocubsgo'. use 'sudo' for root.
cirros login:

[Moha (mohadev)]

I had the same issue: Instances took IP, but not hostname and key pair. The error in instances logs:

    checking http://169.254.169.254/2009-04-04/instance-id
    failed 1/20: up 0.78. request failed
    failed 2/20: up 12.78. request failed
    failed 3/20: up 24.79. request failed
    ...

It got solved by adding `--config-drive true` to the server creation command, or, `force_config_drive = true` into the nova.conf file on the compute node.