Ironic pxe tftp service failed

[swogat pradhan]

Hi,
I am trying to setup openstack wallaby using the repo : centos-release-openstack-wallaby on top of centos 8 stream.

I have deployed undercloud but the service ironic_pxe_tftp is not starting up. Previously the undercloud was failing but now the undercloud is deployed successfylly but the service is not comming up.

Error log from ironic_pxe_tftp:

[root@undercloud ~]# podman logs 23427d845098
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory
/bin/bash: /usr/sbin/in.tftpd: No such file or directory

My undercloud config:

[DEFAULT]
undercloud_hostname = undercloud.taashee.com
container_images_file = containers-prepare-parameter.yaml
local_ip = 192.168.30.50/24
undercloud_public_host = 192.168.30.39
undercloud_admin_host = 192.168.30.41
undercloud_nameservers = 8.8.8.8
pxe_enabled = true
#undercloud_ntp_servers =
overcloud_domain_name = taashee.com
subnets = ctlplane-subnet
local_subnet = ctlplane-subnet
#undercloud_service_certificate =
generate_service_certificate = true
certificate_generation_ca = local
local_interface = eno3
inspection_extras = false
undercloud_debug = false
enable_tempest = false
enable_ui = false

[auth]

[ctlplane-subnet]
cidr = 192.168.30.0/24
dhcp_start = 192.168.30.60
dhcp_end = 192.168.30.100
inspection_iprange = 192.168.30.110,192.168.30.150
gateway = 192.168.30.1

[Takashi Kajinami (kajinamit)]

I've replied back in ml but I suspect your are trying to use c9s containers on cs8 host.
Please double check the definition in ContainerPrepareParmaeters and ensure you are using the correct image(in quey, the namespace should be tripleowallaby, iiuc)

[swogat pradhan (swogat)]

i believe that is the issue, the current continer parameters file is trying to pull centos9 images.
i changed the namespace, but i was unable to find the quay.io/tripleowallaby,iiuc in web, honestly i don't know ow to specify that:
(undercloud) [stack@undercloud ~]$ cat containers-prepare-parameter.yaml
# Generated with the following on 2022-07-04T13:53:39.943715
#
# openstack tripleo container image prepare default --local-push-destination --output-env-file containers-prepare-parameter.yaml
#

parameter_defaults:
  ContainerImagePrepare:
  - push_destination: true
    set:
      ceph_alertmanager_image: alertmanager
      ceph_alertmanager_namespace: quay.ceph.io/prometheus
      ceph_alertmanager_tag: v0.16.2
      ceph_grafana_image: grafana
      ceph_grafana_namespace: quay.ceph.io/app-sre
      ceph_grafana_tag: 6.7.4
      ceph_image: daemon
      ceph_namespace: quay.io/ceph
      ceph_node_exporter_image: node-exporter
      ceph_node_exporter_namespace: quay.ceph.io/prometheus
      ceph_node_exporter_tag: v0.17.0
      ceph_prometheus_image: prometheus
      ceph_prometheus_namespace: quay.ceph.io/prometheus
      ceph_prometheus_tag: v2.7.2
      ceph_tag: v6.0.4-stable-6.0-pacific-centos-8-x86_64
      name_prefix: openstack-
      name_suffix: ''
      namespace: quay.io/tripleowallaby,iiuc
      neutron_driver: ovn
      rhel_containers: false
      tag: current-tripleo
    tag_from_label: rdo_version

Is this how to specify it?

[Takashi Kajinami (kajinamit)]

(pasting here for record)

Can you try

 namespace: quay.io/tripleowallaby

instead ?

[Takashi Kajinami (kajinamit)]

If you are using tripleowallaby then it should be correct. For wallaby we use a separate org which is tripleowallabycentos9 for CentOS 9 container images.

What I'd suspect next is that you are using an old tripleo-heat-templates which does not contain
https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809213 .

We've removed tftp-server from ironic-conductor image by https://review.opendev.org/c/openstack/tripleo-common/+/812690 so the latest container does not include the binary.

[swogat pradhan (swogat)]

This is a fresh installation i did yesterday. Added the centos-release-openstack-wallaby and upgraded the system as well.
Now i am not sure how i can upgrade the tripleo heat templates.

Is the quay.io/tripleowallaby updated with ceontos9 containers somehow?

[Takashi Kajinami (kajinamit)]

It seems we haven't made wallaby release for a while and the patch I mentioned is not included in the last release.
Which version of tripleo-heat-templates is installed ? I guess you are using RDO package which provides only released content and we might need to use the test repos or create a new release.

[swogat pradhan (swogat)]

Please find the versions of the packages installed:

[root@undercloud ~]# rpm -qa | grep tripleo
ansible-role-tripleo-modify-image-1.2.1-1.el8.noarch
openstack-tripleo-validations-14.2.1-1.el8.noarch
openstack-tripleo-common-15.4.0-1.el8.noarch
ansible-tripleo-ipa-0.2.2-1.el8.noarch
openstack-tripleo-common-containers-15.4.0-1.el8.noarch
tripleo-ansible-3.3.0-1.el8.noarch
ansible-tripleo-ipsec-10.0.1-1.el8.noarch
openstack-tripleo-heat-templates-14.3.0-1.el8.noarch
python3-tripleo-common-15.4.0-1.el8.noarch
python3-tripleoclient-16.4.0-1.el8.noarch
puppet-tripleo-14.2.2-1.el8.noarch

I used the centos-release-openstack-wallaby repo

[Takashi Kajinami (kajinamit)]

So the output indicates that you are using 14.3.0 but the commit was merged after that.
AFAIK centos repo provides only released version.

If that is for testing then I'd recommend you try RDO repository.
 https://trunk.rdoproject.org/centos8-wallaby/

See delorean-deps.repo and curent-passed-ci/delorean.repo .

This week we are trying to pull a few backports to wallaby to start some testing, so I'll check with the team whether we can create a release early next week.