Activity log for bug #1950639

Date Who What changed Old value New value Message
2021-11-11 12:07:09 Alexey Kashavkin bug added bug
2021-11-11 12:10:25 Alexey Kashavkin description Description =========== Overcloud nodes cannot switch to available state after introspection. This is because ironic-neutron-agents aren't created. 2021-11-11 14:43:54.500696 | 005056b2-1d0b-2767-891c-00000000000f | TASK | Make nodes available 2021-11-11 14:44:56.370100 | 005056b2-1d0b-2767-891c-00000000000f | FATAL | Make nodes available | localhost | error={"changed": false, "msg": "Timeout waiting for node 9449a890-73ec-4a23-8581-a16ce8bfce89 to have bridge_mappings set in the ironic-neutron-agent entry"} 2021-11-11 14:44:56.371389 | 005056b2-1d0b-2767-891c-00000000000f | TIMING | Make nodes available | localhost | 0:01:02.049977 | 61.87s Steps to reproduce ================== I everything do by TripleO documentation - https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/deployment/install_undercloud.html Environment =========== 1. Wallaby release 2. Undercloud node in VM with Centos 8.4.2105 Logs & Configs ============== In the ironic-neutron-agent logs: 2021-11-11 14:07:41.319 245813 WARNING keystoneauth.discover [req-f3b069d1-5a67-4ff8-a837-3f1673960191 - - - - -] Failed to contact the endpoint at https://192.168.24.2:13385 for discovery. Fallback to using that endpoint as the base url. 2021-11-11 14:07:41.332 245813 WARNING keystoneauth.discover [req-f3b069d1-5a67-4ff8-a837-3f1673960191 - - - - -] Failed to contact the endpoint at https://192.168.24.2:13385 for discovery. Fallback to using that endpoint as the base url. 2021-11-11 14:07:41.333 245813 ERROR networking_baremetal.ironic_client [req-f3b069d1-5a67-4ff8-a837-3f1673960191 - - - - -] Ironic API might not be running, failed to establish a connection with ironic, reason: The baremetal service for 192.168.24.3:regionOne exists but does not have any supported versions.. Retrying ...: openstack.exceptions.NotSupported: The baremetal service for 192.168.24.3:regionOne exists but does not have any supported versions. In HAProxy logs: Nov 11 13:34:17 tripleo-undercloud-1 haproxy[12]: 192.168.24.2:38340 [11/Nov/2021:13:34:17.217] ironic/1: SSL handshake failure Nov 11 13:34:17 tripleo-undercloud-1 haproxy[12]: 192.168.24.2:38342 [11/Nov/2021:13:34:17.228] ironic/1: SSL handshake failure In template for ironic-neutron-agent from kolla-ansible, I saw option - cafile (https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/neutron/templates/ironic_neutron_agent.ini.j2#L11) and manually added them to ironic-neutron-agent.ini: (undercloud) [stack@tripleo-undercloud-1 ~]$ sudo podman exec ironic_neutron_agent cat etc/neutron/plugins/ml2/ironic_neutron_agent.ini [ironic] auth_type=password auth_url=http://192.168.24.3:5000 username=ironic password=S4sKbc7obrtWBgVhxpPGkk08v project_domain_name=Default project_name=service user_domain_name=Default region_name=regionOne cafile=/etc/pki/tls/certs/ca-bundle.trust.crt <--- After that, the ironic-neutron-agent began to work correctly and was created for overcloud nodes: (undercloud) [stack@tripleo-undercloud-1 ~]$ openstack network agent list -c Host -c Binary -c 'Agent Type' --sort-column Binary +--------------------+--------------------------------------+---------------------------+ | Agent Type | Host | Binary | +--------------------+--------------------------------------+---------------------------+ | Baremetal Node | 314ceed1-6412-4f1c-9c9b-1168c0873720 | ironic-neutron-agent | | Baremetal Node | 9449a890-73ec-4a23-8581-a16ce8bfce89 | ironic-neutron-agent | | DHCP agent | tripleo-undercloud-1.localdomain | neutron-dhcp-agent | | L3 agent | tripleo-undercloud-1.localdomain | neutron-l3-agent | | Open vSwitch agent | tripleo-undercloud-1.localdomain | neutron-openvswitch-agent | +--------------------+--------------------------------------+---------------------------+ Description =========== Overcloud nodes cannot switch to available state after introspection. This is because ironic-neutron-agents aren't created. 2021-11-11 14:43:54.500696 | 005056b2-1d0b-2767-891c-00000000000f | TASK | Make nodes available 2021-11-11 14:44:56.370100 | 005056b2-1d0b-2767-891c-00000000000f | FATAL | Make nodes available | localhost | error={"changed": false, "msg": "Timeout waiting for node 9449a890-73ec-4a23-8581-a16ce8bfce89 to have bridge_mappings set in the ironic-neutron-agent entry"} 2021-11-11 14:44:56.371389 | 005056b2-1d0b-2767-891c-00000000000f | TIMING | Make nodes available | localhost | 0:01:02.049977 | 61.87s Steps to reproduce ================== I everything do by TripleO documentation - https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/deployment/install_undercloud.html Environment =========== 1. Wallaby release 2. Undercloud node in VM with Centos 8.4.2105 Logs & Configs ============== In the ironic-neutron-agent logs: 2021-11-11 14:07:41.319 245813 WARNING keystoneauth.discover [req-f3b069d1-5a67-4ff8-a837-3f1673960191 - - - - -] Failed to contact the endpoint at https://192.168.24.2:13385 for discovery. Fallback to using that endpoint as the base url. 2021-11-11 14:07:41.332 245813 WARNING keystoneauth.discover [req-f3b069d1-5a67-4ff8-a837-3f1673960191 - - - - -] Failed to contact the endpoint at https://192.168.24.2:13385 for discovery. Fallback to using that endpoint as the base url. 2021-11-11 14:07:41.333 245813 ERROR networking_baremetal.ironic_client [req-f3b069d1-5a67-4ff8-a837-3f1673960191 - - - - -] Ironic API might not be running, failed to establish a connection with ironic, reason: The baremetal service for 192.168.24.3:regionOne exists but does not have any supported versions.. Retrying ...: openstack.exceptions.NotSupported: The baremetal service for 192.168.24.3:regionOne exists but does not have any supported versions. In HAProxy logs: Nov 11 13:34:17 tripleo-undercloud-1 haproxy[12]: 192.168.24.2:38340 [11/Nov/2021:13:34:17.217] ironic/1: SSL handshake failure Nov 11 13:34:17 tripleo-undercloud-1 haproxy[12]: 192.168.24.2:38342 [11/Nov/2021:13:34:17.228] ironic/1: SSL handshake failure In template for ironic-neutron-agent from kolla-ansible, I saw option - cafile (https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/neutron/templates/ironic_neutron_agent.ini.j2#L11) and manually added them to ironic-neutron-agent.ini: (undercloud) [stack@tripleo-undercloud-1 ~]$ sudo podman exec ironic_neutron_agent cat etc/neutron/plugins/ml2/ironic_neutron_agent.ini [ironic] auth_type=password auth_url=http://192.168.24.3:5000 username=ironic password=S4sKbc7obrtWBgVhxpPGkk08v project_domain_name=Default project_name=service user_domain_name=Default region_name=regionOne cafile=/etc/pki/tls/certs/ca-bundle.trust.crt <--- After that, the ironic-neutron-agent began to work correctly and was created for overcloud nodes: (undercloud) [stack@tripleo-undercloud-1 ~]$ openstack network agent list -c Host -c Binary -c 'Agent Type' --sort-column Binary -f json [ { "Agent Type": "Baremetal Node", "Host": "314ceed1-6412-4f1c-9c9b-1168c0873720", "Binary": "ironic-neutron-agent" }, { "Agent Type": "Baremetal Node", "Host": "9449a890-73ec-4a23-8581-a16ce8bfce89", "Binary": "ironic-neutron-agent" }, { "Agent Type": "DHCP agent", "Host": "tripleo-undercloud-1.localdomain", "Binary": "neutron-dhcp-agent" }, { "Agent Type": "L3 agent", "Host": "tripleo-undercloud-1.localdomain", "Binary": "neutron-l3-agent" }, { "Agent Type": "Open vSwitch agent", "Host": "tripleo-undercloud-1.localdomain", "Binary": "neutron-openvswitch-agent" } ]