Unfortunately we have no log's to see what happened in the booted ramdisk. We just know it booted but never called back to ironic-inspector.
Would it be possible to start dumping 'openstack console log show' for the OVB nodes in the cloud tenant once inspection has started and have multiple timestamped logs collected?
Looking at periodic- tripleo- ci-centos- 8-ovb-3ctlr_ 1comp-featurese t035-train: /review. rdoproject. org/zuul/ build/657be5515 b6543488de53e03 2c03d5aa
https:/
The first attempt to introspect fails, on retry it succedes.
Initial attempt:
DHCP is OK: br-ctlplane) fa:16:3e:52:8f:ef br-ctlplane) 192.168.24.100 fa:16:3e:52:8f:ef br-ctlplane) fa:16:3e:52:8f:ef br-ctlplane) 192.168.24.100 fa:16:3e:52:8f:ef br-ctlplane) 192.168.24.100 fa:16:3e:52:8f:ef br-ctlplane) 192.168.24.100 fa:16:3e:52:8f:ef br-ctlplane) fa:16:3e:58:d7:73 br-ctlplane) 192.168.24.101 fa:16:3e:58:d7:73 br-ctlplane) fa:16:3e:58:d7:73 br-ctlplane) 192.168.24.101 fa:16:3e:58:d7:73 br-ctlplane) 192.168.24.101 fa:16:3e:58:d7:73 br-ctlplane) 192.168.24.101 fa:16:3e:58:d7:73 br-ctlplane) fa:16:3e:d2:e2:f7 br-ctlplane) 192.168.24.102 fa:16:3e:d2:e2:f7 br-ctlplane) fa:16:3e:41:08:50 br-ctlplane) 192.168.24.103 fa:16:3e:41:08:50 br-ctlplane) fa:16:3e:d2:e2:f7 br-ctlplane) 192.168.24.102 fa:16:3e:d2:e2:f7 br-ctlplane) fa:16:3e:41:08:50 br-ctlplane) 192.168.24.103 fa:16:3e:41:08:50 br-ctlplane) 192.168.24.102 fa:16:3e:d2:e2:f7 br-ctlplane) 192.168.24.102 fa:16:3e:d2:e2:f7 br-ctlplane) fa:16:3e:41:08:50 br-ctlplane) 192.168.24.103 fa:16:3e:41:08:50 br-ctlplane) 192.168.24.102 fa:16:3e:d2:e2:f7 br-ctlplane) 192.168.24.102 fa:16:3e:d2:e2:f7 br-ctlplane) 192.168.24.102 fa:16:3e:d2:e2:f7 br-ctlplane) 192.168.24.102 fa:16:3e:d2:e2:f7 br-ctlplane) 192.168.24.102 fa:16:3e:d2:e2:f7 br-ctlplane) 192.168.24.102 fa:16:3e:d2:e2:f7 br-ctlplane) 192.168.24.103 fa:16:3e:41:08:50 br-ctlplane) 192.168.24.103 fa:16:3e:41:08:50 br-ctlplane) fa:16:3e:52:8f:ef br-ctlplane) 192.168.24.100 fa:16:3e:52:8f:ef br-ctlplane) 192.168.24.100 fa:16:3e:52:8f:ef br-ctlplane) 192.168.24.100 fa:16:3e:52:8f:ef br-ctlplane) fa:16:3e:58:d7:73 br-ctlplane) 192.168.24.101 fa:16:3e:58:d7:73 br-ctlplane) 192.168.24.101 fa:16:3e:58:d7:73 br-ctlplane) 192.168.24.101 fa:16:3e:58:d7:73
Oct 18 20:33:24 dnsmasq-dhcp[7]: DHCPDISCOVER(
Oct 18 20:33:24 dnsmasq-dhcp[7]: DHCPOFFER(
Oct 18 20:33:24 dnsmasq-dhcp[7]: DHCPDISCOVER(
Oct 18 20:33:24 dnsmasq-dhcp[7]: DHCPOFFER(
Oct 18 20:33:24 dnsmasq-dhcp[7]: DHCPREQUEST(
Oct 18 20:33:24 dnsmasq-dhcp[7]: DHCPACK(
Oct 18 20:33:28 dnsmasq-dhcp[7]: DHCPDISCOVER(
Oct 18 20:33:28 dnsmasq-dhcp[7]: DHCPOFFER(
Oct 18 20:33:28 dnsmasq-dhcp[7]: DHCPDISCOVER(
Oct 18 20:33:28 dnsmasq-dhcp[7]: DHCPOFFER(
Oct 18 20:33:28 dnsmasq-dhcp[7]: DHCPREQUEST(
Oct 18 20:33:28 dnsmasq-dhcp[7]: DHCPACK(
Oct 18 20:33:46 dnsmasq-dhcp[7]: DHCPDISCOVER(
Oct 18 20:33:46 dnsmasq-dhcp[7]: DHCPOFFER(
Oct 18 20:33:49 dnsmasq-dhcp[7]: DHCPDISCOVER(
Oct 18 20:33:49 dnsmasq-dhcp[7]: DHCPOFFER(
Oct 18 20:33:49 dnsmasq-dhcp[7]: DHCPDISCOVER(
Oct 18 20:33:49 dnsmasq-dhcp[7]: DHCPOFFER(
Oct 18 20:33:49 dnsmasq-dhcp[7]: DHCPDISCOVER(
Oct 18 20:33:49 dnsmasq-dhcp[7]: DHCPOFFER(
Oct 18 20:33:49 dnsmasq-dhcp[7]: DHCPREQUEST(
Oct 18 20:33:49 dnsmasq-dhcp[7]: DHCPACK(
Oct 18 20:33:49 dnsmasq-dhcp[7]: DHCPDISCOVER(
Oct 18 20:33:49 dnsmasq-dhcp[7]: DHCPOFFER(
Oct 18 20:33:49 dnsmasq-dhcp[7]: DHCPREQUEST(
Oct 18 20:33:49 dnsmasq-dhcp[7]: DHCPACK(
Oct 18 20:33:49 dnsmasq-dhcp[7]: DHCPREQUEST(
Oct 18 20:33:49 dnsmasq-dhcp[7]: DHCPACK(
Oct 18 20:33:49 dnsmasq-dhcp[7]: DHCPREQUEST(
Oct 18 20:33:49 dnsmasq-dhcp[7]: DHCPACK(
Oct 18 20:33:49 dnsmasq-dhcp[7]: DHCPREQUEST(
Oct 18 20:33:49 dnsmasq-dhcp[7]: DHCPACK(
Oct 18 20:33:51 dnsmasq-dhcp[7]: DHCPDISCOVER(
Oct 18 20:33:51 dnsmasq-dhcp[7]: DHCPOFFER(
Oct 18 20:33:51 dnsmasq-dhcp[7]: DHCPREQUEST(
Oct 18 20:33:51 dnsmasq-dhcp[7]: DHCPACK(
Oct 18 20:33:54 dnsmasq-dhcp[7]: DHCPDISCOVER(
Oct 18 20:33:54 dnsmasq-dhcp[7]: DHCPOFFER(
Oct 18 20:33:54 dnsmasq-dhcp[7]: DHCPREQUEST(
Oct 18 20:33:54 dnsmasq-dhcp[7]: DHCPACK(
Since we two DHCPACK's per node, I belive both PXE boot and ironic-python-agent did succesfully DHCP.
HTTP download ironic-python-agent OK: 2021:20: 33:38 +0000] "GET /inspector.ipxe HTTP/1.1" 200 385 "-" "iPXE/1.0.0+ (c63ef)" 2021:20: 33:38 +0000] "GET /agent.kernel HTTP/1.1" 200 10218632 "-" "iPXE/1.0.0+ (c63ef)" 2021:20: 33:38 +0000] "GET /agent.ramdisk HTTP/1.1" 200 483681924 "-" "iPXE/1.0.0+ (c63ef)" 2021:20: 33:41 +0000] "GET /inspector.ipxe HTTP/1.1" 200 385 "-" "iPXE/1.0.0+ (c63ef)" 2021:20: 33:41 +0000] "GET /agent.kernel HTTP/1.1" 200 10218632 "-" "iPXE/1.0.0+ (c63ef)" 2021:20: 33:42 +0000] "GET /agent.ramdisk HTTP/1.1" 200 483681924 "-" "iPXE/1.0.0+ (c63ef)" 2021:20: 33:59 +0000] "GET /inspector.ipxe HTTP/1.1" 200 385 "-" "iPXE/1.0.0+ (c63ef)" 2021:20: 33:59 +0000] "GET /agent.kernel HTTP/1.1" 200 10218632 "-" "iPXE/1.0.0+ (c63ef)" 2021:20: 33:59 +0000] "GET /inspector.ipxe HTTP/1.1" 200 385 "-" "iPXE/1.0.0+ (c63ef)" 2021:20: 33:59 +0000] "GET /agent.kernel HTTP/1.1" 200 10218632 "-" "iPXE/1.0.0+ (c63ef)" 2021:20: 33:59 +0000] "GET /agent.ramdisk HTTP/1.1" 200 483681924 "-" "iPXE/1.0.0+ (c63ef)" 2021:20: 34:00 +0000] "GET /agent.ramdisk HTTP/1.1" 200 483681924 "-" "iPXE/1.0.0+ (c63ef)"
192.168.24.100 - - [18/Oct/
192.168.24.100 - - [18/Oct/
192.168.24.100 - - [18/Oct/
192.168.24.101 - - [18/Oct/
192.168.24.101 - - [18/Oct/
192.168.24.101 - - [18/Oct/
192.168.24.102 - - [18/Oct/
192.168.24.102 - - [18/Oct/
192.168.24.103 - - [18/Oct/
192.168.24.103 - - [18/Oct/
192.168.24.102 - - [18/Oct/
192.168.24.103 - - [18/Oct/
Unfortunately we have no log's to see what happened in the booted ramdisk. We just know it booted but never called back to ironic-inspector.
Would it be possible to start dumping 'openstack console log show' for the OVB nodes in the cloud tenant once inspection has started and have multiple timestamped logs collected?