Rocky prechecks - template error while templating string: no filter named 'search'

[alpha23]

I received the following failed task:

TASK [prechecks : Checking empty passwords in passwords.yml. Run kolla-genpwd if this task fails] **********************************************************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'result.stdout | regex_replace('(.*ssh_key.*)', '') | search(\":\")' failed. The error was: template error while templating string: no filter named 'search'. String: {% if result.stdout | regex_replace('(.*ssh_key.*)', '') | search(\":\") %} True {% else %} False {% endif %}"}

While it appears the filters changed in Ansible 2.9 (which I tested on), I also tested using Ansible 2.7.0, 2.8.0 and 2.8.5.

Per the Rocky Quick Start page, it merely states the requirement of Ansible >= 2.4 which coincides with ansible/roles/prechecks/vars/main.yml.

Using:
Ubuntu 18.04.4 LTS, 4.15.0-88-generic
Docker: 18.06.1-ce

[alpha23 (alpha23)]

This was resolved by correcting the virtual environment which was using Ansible 2.9.9 and not 2.8.5, etc. However, the issue still remains that for Rocky and Ansible 2.9+, the above error is exhibited and both the http documents and kolla-ansible prechecks (which verifies if Ansible is 2.4+) are incorrect and need to be corrected.

[Radosław Piliszek (yoctozepto)]

Rocky is extended maintenance and it was not meant to support Ansible 2.9. Please propose a patch if you know one. We can also mention the upper bound in the docs.

[Mark Goddard (mgoddard)]

I think we can't support both 2.4 and 2.9

[Radosław Piliszek (yoctozepto)]

Yeah, we can't. Hence this is docs to update in fact.

[alpha23 (alpha23)]

@Radoslaw, this is also a code issue as noted above.

"Rocky is extended maintenance and it was not meant to support Ansible 2.9." How is anyone supposed to know this? It is not in the code and it is not documented.

The issue is not just with this one issue but with the slew of undocumented and/or buggy code in numerous projects which in many cases is just due to lazy development and seems to be prevalent with many OS projects. This one issue then gets coupled with thousands of other nuances. Are there any regression tests with kolla-ansible and how often are these run?

Rocky has been out for ~18 months now and is now on 7.2.1. I couldn't have been the first person to see this undocumented issue and probably won't be the last. While it took me maybe an hour to resolve the issue, it will equivalently take hundreds, if not thousands, of others approximately the same leading to a lot of wasted time all over a 5 minute requirement implementation and a 30 second inclusion in the docs.

[Mark Goddard (mgoddard)]

@alpha23, we added a max version check in Ussuri for this reason, and document lower and upper limits. It should have been there sooner, but wasn't.

Radoslaw is suggesting we (or better, you) update the docs to account for this where it isn't mentioned.

As a general comment, we appreciate patches more than complaints.

[alpha23 (alpha23)]

@Mgoddard, thank you for adding to Ussari; will this be documented in Stein and Train as well to save others from the same issue?

Also, are there any regression tests with kolla-ansible and how often are these run?

"As a general comment, we appreciate patches more than complaints." While I understand your statement, the attitude that "someone ElSE can just create a patch if it doesn't work" has lead to poor development and documentation practices where developers do not test and take no care in the code that they are churning out. What is the test code coverage ratio for kolla-ansible? There are currently 413 open bugs in kolla-ansible alone. This attitude is defeating your mission statement which in part reads "...[Kolla-Ansible] permits operators with minimal experience to deploy OpenStack quickly.." Moreover, it is unreasonable to expect every user, especially those with "minimal experience," to become an expert and propose patches for every Openstack project. I'm currently in my 2nd day of upgrading from Queens to Rocky and some of the issues that I'm running into appear to be over a year old. Thus the reason that I've held off upgrading from Queens for at least a year after Rocky was released hoping to avoid buggy code. Unfortunately, this has lead to our entire Openstack deployment being down and substantial downtime and loss of productivity for a number of users including me.

[Mark Goddard (mgoddard)]

I recognise that it's frustrating when software breaks or documentation is wrong. Every patch that gets accepted must pass our CI testing. The coverage isn't perfect, but is improving. Kolla-ansible touches a huge number of different pieces of software, and things do break from time to time. Kolla isn't an open source project, and there isn't a large distro vendor behind it, so it relies on operators to build, test and maintain it. Which they do.

I'll submit a patch to fix the docs for Stein and Rocky.

[Mark Goddard (mgoddard)]

Kolla is an open source project

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/728783

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/728783
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=7b22f394f3392dc63f16cb6a195309f7b0ddf929
Submitter: Zuul
Branch: stable/train

commit 7b22f394f3392dc63f16cb6a195309f7b0ddf929
Author: Mark Goddard <email address hidden>
Date: Mon May 18 09:31:29 2020 +0100

    Document and test maximum supported version of Ansible

    The Train release supports up to Ansible 2.9. This patch states this in
    the documentation, and updates our testing to match.

    Change-Id: I83d068a242d2e64b5e971be1549935f87641cf8e
    Closes-Bug: #1878328

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/728898

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/728928

[alpha23 (alpha23)]

@mgoodard, What is the test code coverage ratio? Things are not breaking some of the time but they are breaking regularly as is indicated by the number of open bugs. I'm currently on day 5 of my Queens to Rocky upgrade and I have much more than "minimal experience."

The problem with the statement "Kolla is an open source project" is that this attitude pervades the community so that sound software development practices such as unit tests, code coverage tests, quality gates, etc. all are overlooked because "it is open source." Open source does not mean that sound code practices should be abandoned. It appears though is exactly the case.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/728928
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=02f9c2a2cd3537748a9b294927b6f7623fae3331
Submitter: Zuul
Branch: stable/rocky

commit 02f9c2a2cd3537748a9b294927b6f7623fae3331
Author: Mark Goddard <email address hidden>
Date: Mon May 18 09:31:29 2020 +0100

    Document maximum supported version of Ansible

    The Rocky release supports up to Ansible 2.8. This patch states this in
    the documentation.

    Change-Id: I83d068a242d2e64b5e971be1549935f87641cf8e
    Closes-Bug: #1878328
    (cherry picked from commit 7b22f394f3392dc63f16cb6a195309f7b0ddf929)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/728898
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=fcbcd1e36f8c8c64ec806731657f79fe0d119b7c
Submitter: Zuul
Branch: stable/stein

commit fcbcd1e36f8c8c64ec806731657f79fe0d119b7c
Author: Mark Goddard <email address hidden>
Date: Mon May 18 09:31:29 2020 +0100

    Document and test maximum supported version of Ansible

    The Stein release supports up to Ansible 2.9. This patch states this in
    the documentation, and updates our testing to match.

    Change-Id: I83d068a242d2e64b5e971be1549935f87641cf8e
    Closes-Bug: #1878328
    (cherry picked from commit 7b22f394f3392dc63f16cb6a195309f7b0ddf929)

[Marcin Juszkiewicz (hrw)]

@alpha23 - come, join us. Spend some time on testing upgrades, write what issues you had, fill bugs, discuss them with us etc.

At Linaro we had rocky->stein->train upgrade done recently. Ended with pile of changes landed in Kolla(-ansible) to sort out rough edges and get such upgrades easier for others.

Kolla has 4-6 developers now. Most (all?) of them work on other projects at same time. We added several new CI scenarios during last year as we had reports that something is broken here or there. We can not add hundred CI jobs to cover every possible situation as this would overload CI infrastructure.

And while I understand you waiting with queens->rocky upgrade you should understand that once project goes into EM phase (like rocky did a while ago) there will be less and less people interested with working on it. Especially when issues cover multiple projects.

[alpha23 (alpha23)]

@hrw And therein lies the problem. Inadequate testing and poor development practices. It doesn't matter if the "team" is 1 person or 100 people, proper development practices are designed to ensure that the code that is developed is reliable. There are currently at least 49 bugs in rocky and it is already in the "goodby" phase. That is not reliable.

I propose that you change your mission statement, "...[Kolla-Ansible] permits operators with minimal experience to deploy OpenStack quickly.." because in my experience the current statement is just wishful thinking.

I still have not gotten an answer on what is the test code coverage. Specifically, the test branch code coverage for kolla-ansible? This information should be on github as it is with many projects.

[alpha23 (alpha23)]

Correction, there are not 49 open bugs in rocky but the concern still remains.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible rocky-eol release.