Issue while importing nodes and making them available

[Cédric Jeanneret]

Hello,

It seems the "provide" command is hitting a timeout on my env[1]:

2021-08-23 13:33:46.122226 | 2442014a-9670-7de9-f208-00000000000f | FATAL | Make nodes available | localhost | error={"changed": false, "msg": "Timeout waiting for node 1b82598d-5c0d-447e-b0e8-05c5cc8b2b63 to have bridge_mappings set
in the ironic-neutron-agent entry"}

The task that actually hits this timeout is this one:
https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/ansible_plugins/modules/os_baremetal_provide_node.py#L379-L389

I tried to make that timeout a bit bigger (120 instead of 60), but nope, it's still failing.

After a check on neutron agents, it seems the one we're needing is taking too long to pop in the list.
Before the timeout, there's this list:
+--------------------------------------+--------------------+-------------------------+-------------------+-------+-------+---------------------------+
| ID | Agent Type | Host | Availability Zone | Alive | State | Binary |
+--------------------------------------+--------------------+-------------------------+-------------------+-------+-------+---------------------------+
| 5f390380-74ee-44f3-b3e5-b1cbf18fe142 | Open vSwitch agent | undercloud.mydomain.tld | None | :-) | UP | neutron-openvswitch-agent |
| 7be250d6-b9ae-4a35-8985-5e0c4375d848 | L3 agent | undercloud.mydomain.tld | nova | :-) | UP | neutron-l3-agent |
| ecee547a-d0e1-4605-8dd4-c21a98ebfc4f | DHCP agent | undercloud.mydomain.tld | nova | :-) | UP | neutron-dhcp-agent |
+--------------------------------------+--------------------+-------------------------+-------------------+-------+-------+---------------------------+

We can see we're missing the ironic-neutron-agent one. It appears a bit later though:
+--------------------------------------+--------------------+--------------------------------------+-------------------+-------+-------+---------------------------+
| ID | Agent Type | Host | Availability Zone | Alive | State | Binary |
+--------------------------------------+--------------------+--------------------------------------+-------------------+-------+-------+---------------------------+
| 5f390380-74ee-44f3-b3e5-b1cbf18fe142 | Open vSwitch agent | undercloud.mydomain.tld | None | :-) | UP | neutron-openvswitch-agent |
| 7be250d6-b9ae-4a35-8985-5e0c4375d848 | L3 agent | undercloud.mydomain.tld | nova | :-) | UP | neutron-l3-agent |
| ad27978d-a635-4054-b798-c45d6d92da3f | Baremetal Node | 1b82598d-5c0d-447e-b0e8-05c5cc8b2b63 | None | :-) | UP | ironic-neutron-agent |
| ecee547a-d0e1-4605-8dd4-c21a98ebfc4f | DHCP agent | undercloud.mydomain.tld | nova | :-) | UP | neutron-dhcp-agent |
+--------------------------------------+--------------------+--------------------------------------+-------------------+-------+-------+---------------------------+

Seems that 120s isn't long enough, but if we have to raise that timeout that much, it probably means there's "something" really slow in the whole chain.

It was working fine on the very same hardware last week.
The UC node has 12-cores and 27G of RAM (in a virtual machine)
The OC is just 1-controller

TripleO related packages versions:
openstack-tripleo-heat-templates-15.0.1-0.20210823104807.d12028c.el8.noarch
tripleo-ansible-4.0.1-0.20210823130700.75cb8b5.el8.noarch
python3-tripleoclient-17.0.1-0.20210821033245.72cba87.el8.noarch
ansible-tripleo-ipsec-11.0.1-0.20210304160420.b5559c8.el8.noarch
openstack-tripleo-validations-15.0.1-0.20210819182907.d00a7f9.el8.noarch
openstack-tripleo-common-containers-16.1.1-0.20210823075805.1ef4c88.el8.noarch
puppet-tripleo-15.0.1-0.20210823102254.3296269.el8.noarch
ansible-tripleo-ipa-0.2.2-0.20210422191945.9159108.el8.noarch
openstack-tripleo-common-16.1.1-0.20210823075805.1ef4c88.el8.noarch
python3-tripleo-repos-0.1.1-0.20210817121802.041c6d5.el8.noarch
ansible-role-tripleo-modify-image-1.2.3-0.20210820065807.0b9fdcc.el8.noarch
python3-tripleo-common-16.1.1-0.20210823075805.1ef4c88.el8.noarch

The containers were built on Aug-18, and ironic_neutron_agent container has the following neutron-related packages:
python3-neutron-18.1.0-0.20210816102305.7988ab5.el8.noarch
openstack-neutron-ml2-18.1.0-0.20210816102305.7988ab5.el8.noarch
python3-ironic-neutron-agent-4.1.0-0.20210802103610.ae68d69.el8.noarch
python3-neutronclient-7.5.0-0.20210610134047.ae39756.el8.noarch
openstack-neutron-common-18.1.0-0.20210816102305.7988ab5.el8.noarch
openstack-neutron-18.1.0-0.20210816102305.7988ab5.el8.noarch
openstack-neutron-rpc-server-18.1.0-0.20210816102305.7988ab5.el8.noarch
python3-neutron-lib-2.13.0-0.20210805084212.2b46c1f.el8.noarch

Not really sure what data I can provide. Timeline?

We can see in /var/log/containers/neutron/ironic-neutron-agent.log the following lines:
2021-08-23 13:25:15.383 79921 INFO networking_baremetal.agent.ironic_neutron_agent [-] Starting agent networking-baremetal.
2021-08-23 13:25:15.527 79921 INFO networking_baremetal.agent.ironic_neutron_agent [-] Adding member id 56bb8051-e0f1-4e2a-9b1d-6c4ef3bdce05 on host undercloud.mydomain.tld to hashring.
2021-08-23 13:35:15.569 79921 INFO networking_baremetal.agent.ironic_neutron_agent [-] Reporting state for host agent 1b82598d-5c0d-447e-b0e8-05c5cc8b2b63 with new configuration: {'bridge_mappings': {'ctlplane': 'yes'}, 'log_agent_heartbeats': False}
2021-08-23 13:35:15.570 79921 DEBUG networking_baremetal.agent.ironic_neutron_agent [-] Reporting state for host: 1b82598d-5c0d-447e-b0e8-05c5cc8b2b63 with configuration: {'bridge_mappings': {'ctlplane': 'yes'}, 'log_agent_heartbeats': False} _report_state /usr/lib/python3.6/site-packages/networking_baremetal/agent/ironic_neutron_agent.py:241
2021-08-23 13:40:15.595 79921 DEBUG networking_baremetal.agent.ironic_neutron_agent [-] Reporting state for host: 1b82598d-5c0d-447e-b0e8-05c5cc8b2b63 with configuration: {'bridge_mappings': {'ctlplane': 'yes'}, 'log_agent_heartbeats': False} _report_state /usr/lib/python3.6/site-packages/networking_baremetal/agent/ironic_neutron_agent.py:241

It seems the relevant line (Reporting state for host agent 1b82598d-5c0d-447e-b0e8-05c5cc8b2b63 with new configuration) comes almost 2 minutes after the timeout error reported by `openstack overcloud node provision'...

Maybe a hint?

Cheers,

C.

[1] https://github.com/cjeanner/tripleo-lab

[Cédric Jeanneret (cjeanner)]

Oh, another thing:

(undercloud) [CentOS-8 - stack@undercloud ~]$ openstack network agent show ad27978d-a635-4054-b798-c45d6d92da3f
+-------------------+-------------------------------------------------------------------------+
| Field | Value |
+-------------------+-------------------------------------------------------------------------+
| admin_state_up | UP |
| agent_type | Baremetal Node |
| alive | :-) |
| availability_zone | None |
| binary | ironic-neutron-agent |
| configuration | {'bridge_mappings': {'ctlplane': 'yes'}, 'log_agent_heartbeats': False} |
| created_at | 2021-08-23 13:35:15 |
| description | None |
| ha_state | None |
| host | 1b82598d-5c0d-447e-b0e8-05c5cc8b2b63 |
| id | ad27978d-a635-4054-b798-c45d6d92da3f |
| last_heartbeat_at | 2021-08-23 13:45:15 |
| name | None |
| resources_synced | None |
| started_at | 2021-08-23 13:35:15 |
| topic | N/A |
+-------------------+-------------------------------------------------------------------------+

We can see it's, indeed, started pretty late, compared to the failure timeline :(.

[Harald Jensås (harald-jensas)]

2021-08-23 13:25:15.527 79921 INFO networking_baremetal.agent.ironic_neutron_agent [-] Adding member id 56bb8051-e0f1-4e2a-9b1d-6c4ef3bdce05 on host undercloud.mydomain.tld to hashring.

... At this point the ironic-neutron-agent will do a "baremetal port list", and adding/updating ironic node agents in neutron every 30s, here is the code[1].

The timeout waiting for node 1b82598d-5c0d-447e-b0e8-05c5cc8b2b63 bridge mappings was timestamped 13:33:46. The agent reports it 2 minutes later, indicating something is very slow?

2021-08-23 13:35:15.569 79921 INFO networking_baremetal.agent.ironic_neutron_agent [-] Reporting state for host agent 1b82598d-5c0d-447e-b0e8-05c5cc8b2b63 with new configuration: {'bridge_mappings': {'ctlplane': 'yes'}, 'log_agent_heartbeats': False}

Can you attach ironic logs?
- When did node 1b82598d-5c0d-447e-b0e8-05c5cc8b2b63 get enrolled in ironic?
- Do we see the "baremteal port list" calls from the agent in Ironic API logs? Are they very slow?

[1] https://opendev.org/openstack/networking-baremetal/src/branch/master/networking_baremetal/agent/ironic_neutron_agent.py#L207

[Cédric Jeanneret (cjeanner)]

Hey Harald,

Thanks for jumping in!

sooo. If I take the ironic log from yesterday:
ironic-conductor.log.1

I can see first mention of that node as follow:
2021-08-23 13:31:37.049 7 DEBUG ironic_lib.json_rpc.server [req-493790d9-3519-4130-8ad9-2ef91cd6d4f7 e707cf08f0fb46f489e760eb51f9d698 3c1bf87391c64bd28498e39375cdcaf4 - default default] RPC create_node with {'node_obj': {'ironic_object.name': 'Node', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.36', 'ironic_object.data': {'uuid': '1b82598d-5c0d-447e-b0e8-05c5cc8b2b63', 'name': 'oc0-controller-0', 'chassis_id': None, 'driver': 'ipmi', 'driver_info': {'deploy_kernel': 'file:///var/lib/ironic/httpboot/agent.kernel', 'rescue_kernel': 'file:///var/lib/ironic/httpboot/agent.kernel', 'deploy_ramdisk': 'file:///var/lib/ironic/httpboot/agent.ramdisk', 'rescue_ramdisk': 'file:///var/lib/ironic/httpboot/agent.ramdisk', 'ipmi_username': 'ADMIN', 'ipmi_password': '***', 'ipmi_address': '192.168.122.1', 'ipmi_port': '15566'}, 'properties': {'cpus': '12', 'memory_mb': '16384', 'local_gb': '100', 'cpu_arch': 'x86_64', 'capabilities': 'boot_option:local,node:0-controller-0'}, 'conductor_group': '', 'provision_state': 'enroll', 'resource_class': 'baremetal'}, 'ironic_object.changes': ['driver', 'provision_state', 'properties', 'uuid', 'conductor_group', 'resource_class', 'driver_info', 'name', 'chassis_id']}, 'context': {'user': 'e707cf08f0fb46f489e760eb51f9d698', 'tenant': '3c1bf87391c64bd28498e39375cdcaf4', 'system_scope': None, 'project': '3c1bf87391c64bd28498e39375cdcaf4', 'domain': None, 'user_domain': 'default', 'project_domain': 'default', 'is_admin': False, 'read_only': False, 'show_deleted': False, 'auth_token': '***', 'request_id': 'req-493790d9-3519-4130-8ad9-2ef91cd6d4f7', 'global_request_id': None, 'resource_uuid': None, 'roles': ['reader', 'admin', 'member', '_member_'], 'user_identity': 'e707cf08f0fb46f489e760eb51f9d698 3c1bf87391c64bd28498e39375cdcaf4 - default default', 'is_admin_project': True}} _handle_requests /usr/lib/python3.6/site-packages/ironic_lib/json_rpc/server.py:279

It looks like the actual addition, doesn't it? So timestamp mark is 2 minutes before the node "provide" crash.

Regarding the port list (that's the line in ironic-neutron-agent.log, right?), this (partial) log might help a bit:

2021-08-23 13:25:15.383 79921 INFO networking_baremetal.agent.ironic_neutron_agent [-] Starting agent networking-baremetal.
2021-08-23 13:25:15.527 79921 INFO networking_baremetal.agent.ironic_neutron_agent [-] Adding member id 56bb8051-e0f1-4e2a-9b1d-6c4ef3bdce05 on host undercloud.mydomain.tld to hashring.
2021-08-23 13:35:15.569 79921 INFO networking_baremetal.agent.ironic_neutron_agent [-] Reporting state for host agent 1b82598d-5c0d-447e-b0e8-05c5cc8b2b63 with new configuration: {'bridge_mappings': {'ctlplane': 'yes'}, 'log_agent_heartbeats': False} ...

[Harald Jensås (harald-jensas)]

2021-08-23 13:25:15.527 79921 INFO networking_baremetal.agent.ironic_neutron_agent [-] Adding member id 56bb8051-e0f1-4e2a-9b1d-6c4ef3bdce05 on host undercloud.mydomain.tld to hashring.

... At this point the ironic-neutron-agent will do a "baremetal port list", and adding/updating ironic node agents in neutron every 30s, here is the code[1].

The timeout waiting for node 1b82598d-5c0d-447e-b0e8-05c5cc8b2b63 bridge mappings was timestamped 13:33:46. The agent reports it 2 minutes later, indicating something is very slow?

2021-08-23 13:35:15.569 79921 INFO networking_baremetal.agent.ironic_neutron_agent [-] Reporting state for host agent 1b82598d-5c0d-447e-b0e8-05c5cc8b2b63 with new configuration: {'bridge_mappings': {'ctlplane': 'yes'}, 'log_agent_heartbeats': False}

Can you attach ironic logs?
- When did node 1b82598d-5c0d-447e-b0e8-05c5cc8b2b63 get enrolled in ironic?
- Do we see the "baremteal port list" calls from the agent in Ironic API logs? Are they very slow?

[1] https://opendev.org/openstack/networking-baremetal/src/branch/master/networking_baremetal/agent/ironic_neutron_agent.py#L207

[Harald Jensås (harald-jensas)]

The issue is that the agent is reporting at 5 minute intervals, not 30 seconds.

2021-08-26 12:30:41.101 495681 INFO networking_baremetal.agent.ironic_neutron_agent [-] Starting agent networking-baremetal.
2021-08-26 12:30:41.137 495681 INFO networking_baremetal.agent.ironic_neutron_agent [-] Adding member id c68f1490-4252-4c74-be59-3a02f5b6bce3 on host undercloud.mydomain.tld to hashring.
2021-08-26 12:35:41.130 495681 DEBUG networking_baremetal.agent.ironic_neutron_agent [-] Listing Ironic ports. _report_state /usr/lib/python3.6/site-packages/networking_baremetal/agent/ironic_neutron_agent.py:209
2021-08-26 12:35:41.179 495681 DEBUG networking_baremetal.agent.ironic_neutron_agent [-] Found Ironic ports: [] _report_state /usr/lib/python3.6/site-packages/networking_baremetal/agent/ironic_neutron_agent.py:211
2021-08-26 12:40:41.131 495681 DEBUG networking_baremetal.agent.ironic_neutron_agent [-] Listing Ironic ports. _report_state /usr/lib/python3.6/site-packages/networking_baremetal/agent/ironic_neutron_agent.py:209
2021-08-26 12:40:41.187 495681 DEBUG networking_baremetal.agent.ironic_neutron_agent [-] Found Ironic ports: [] _report_state /usr/lib/python3.6/site-packages/networking_baremetal/agent/ironic_neutron_agent.py:211

This change[1] in TripleO changed it to 300s, instead of the neutron default which is 30s.

<hjensas> ksambor: ^^ Any reason to up the report interval from 30s (the default) to 300s? https://opendev.org/openstack/neutron/src/branch/master/neutron/conf/agent/common.py#L112
<hjensas> Tengu: ^ that explains the bug.
<ksambor> hjensas, yah it was to small for bigger/more dense deployments with ovn

It seems the longer interval makes sense for OVN so we should try to find a way to reduce the interval specifically for ironic-neutron-agent instead of reverting it.

https://opendev.org/openstack/tripleo-heat-templates/commit/5ab70af5a681c2281d0a8883b6aaf41f94f0855f

[Harald Jensås (harald-jensas)]

Adding this to: /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/ironic_neutron_agent.ini

  [agent]
  report_interval=30

Makes the agent report at the 30s interval by overriding the report_interval for this networking-baremetal agent only.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/806193

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/806193
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/9574cd4d0f65fd6f7a5a6e04379f8bc4fd6c020c
Submitter: "Zuul (22348)"
Branch: master

commit 9574cd4d0f65fd6f7a5a6e04379f8bc4fd6c020c
Author: Harald Jensås <email address hidden>
Date: Thu Aug 26 15:48:59 2021 +0200

    Add IronicNeutronAgentReportInterval parameter

    Add IronicNeutronAgentReportInterval with a default of
    30 seconds.

    Previously the ironic-neutron-agent used the global report
    interval in neutron. The global interval was increased to
    300s in change: Ib3f7cd9d6c050140a5e8b59adf7fd8f65b12df2f.

    While the 300s interval may make sense for other agents,
    the ironic-neutron-agent can keep reporting at 30s, the
    previous default.

    Closes-Bug: #1940838
    Depends-On: I6fc0eb085c09fb1e24ff09d4f8199d2cf57dd209
    Change-Id: Id77662e9dd466ec8609354c2dbeee8592f599e72

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 15.1.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/828408

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/828557

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/828558

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/828559

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/828408
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/36be5cecad5afb15f2598fd19798e7797453d814
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 36be5cecad5afb15f2598fd19798e7797453d814
Author: Harald Jensås <email address hidden>
Date: Thu Aug 26 15:48:59 2021 +0200

    Add IronicNeutronAgentReportInterval parameter

    Add IronicNeutronAgentReportInterval with a default of
    30 seconds.

    Previously the ironic-neutron-agent used the global report
    interval in neutron. The global interval was increased to
    300s in change: Ib3f7cd9d6c050140a5e8b59adf7fd8f65b12df2f.

    While the 300s interval may make sense for other agents,
    the ironic-neutron-agent can keep reporting at 30s, the
    previous default.

    Conflicts:
      deployment/ironic/ironic-neutron-agent-container-puppet.yaml

    Closes-Bug: #1940838
    Depends-On: https://review.opendev.org/828409
    Change-Id: Id77662e9dd466ec8609354c2dbeee8592f599e72
    (cherry picked from commit 9574cd4d0f65fd6f7a5a6e04379f8bc4fd6c020c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/828558
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/c6ae51e5ce223d376569e3b8f281ff0891d73c55
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit c6ae51e5ce223d376569e3b8f281ff0891d73c55
Author: Harald Jensås <email address hidden>
Date: Thu Aug 26 15:48:59 2021 +0200

    Add IronicNeutronAgentReportInterval parameter

    Add IronicNeutronAgentReportInterval with a default of
    30 seconds.

    Previously the ironic-neutron-agent used the global report
    interval in neutron. The global interval was increased to
    300s in change: Ib3f7cd9d6c050140a5e8b59adf7fd8f65b12df2f.

    While the 300s interval may make sense for other agents,
    the ironic-neutron-agent can keep reporting at 30s, the
    previous default.

    Conflicts:
      deployment/ironic/ironic-neutron-agent-container-puppet.yaml

    Closes-Bug: #1940838
    Depends-On: https://review.opendev.org/828409
    Change-Id: Id77662e9dd466ec8609354c2dbeee8592f599e72
    (cherry picked from commit 9574cd4d0f65fd6f7a5a6e04379f8bc4fd6c020c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/828557
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/b69cdee8df553e51f5834a87a694c7015932b4f0
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit b69cdee8df553e51f5834a87a694c7015932b4f0
Author: Harald Jensås <email address hidden>
Date: Thu Aug 26 15:48:59 2021 +0200

    Add IronicNeutronAgentReportInterval parameter

    Add IronicNeutronAgentReportInterval with a default of
    30 seconds.

    Previously the ironic-neutron-agent used the global report
    interval in neutron. The global interval was increased to
    300s in change: Ib3f7cd9d6c050140a5e8b59adf7fd8f65b12df2f.

    While the 300s interval may make sense for other agents,
    the ironic-neutron-agent can keep reporting at 30s, the
    previous default.

    Conflicts:
      deployment/ironic/ironic-neutron-agent-container-puppet.yaml

    Closes-Bug: #1940838
    Depends-On: https://review.opendev.org/828409
    Change-Id: Id77662e9dd466ec8609354c2dbeee8592f599e72
    (cherry picked from commit 9574cd4d0f65fd6f7a5a6e04379f8bc4fd6c020c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/828559
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/0db8d673d84329c749724a37a962f79ce134a453
Submitter: "Zuul (22348)"
Branch: stable/train

commit 0db8d673d84329c749724a37a962f79ce134a453
Author: Harald Jensås <email address hidden>
Date: Thu Aug 26 15:48:59 2021 +0200

    Add IronicNeutronAgentReportInterval parameter

    Add IronicNeutronAgentReportInterval with a default of
    30 seconds.

    Previously the ironic-neutron-agent used the global report
    interval in neutron. The global interval was increased to
    300s in change: Ib3f7cd9d6c050140a5e8b59adf7fd8f65b12df2f.

    While the 300s interval may make sense for other agents,
    the ironic-neutron-agent can keep reporting at 30s, the
    previous default.

    Conflicts:
      deployment/ironic/ironic-neutron-agent-container-puppet.yaml

    Closes-Bug: #1940838
    Depends-On: https://review.opendev.org/828409
    Change-Id: Id77662e9dd466ec8609354c2dbeee8592f599e72
    (cherry picked from commit 9574cd4d0f65fd6f7a5a6e04379f8bc4fd6c020c)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 12.4.6 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 13.6.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates train-eol release.