zed images can't be build (Ubuntu/CentOS).

[Gaël THEROND]

Building ZED release images following the documentation isn't working.

If you try to use fully public and native way to build the images for the zed release, the process fails with the following bug: https://paste.opendev.org/show/bOGi62PqiJho39qHMGHS/

** HOW TO REPRODUCE **
1°/ - Generate a default kolla-build.conf configuration file.
2°/ - Use Ubuntu or CentOS as base.
3°/ - Run a build on Windows Docker Desktop as for: https://paste.opendev.org/show/bvVAlhovMS89K0FWQyxX/
4°/ - kolla-requirements.txt content: https://paste.opendev.org/show/bhkLkdDbfmfQ3YR53hxW/

This run was also tested on a classical jenkins job over a ubuntu 22.04.4 LTS host with the same result.

** WHAT HAPPENS **
It happens because when building the plugins archive, some images miss the whole item.source dict during the make_an_archive() function call on plugin_am assignation for images plugins archive build.

As a result, because make_an_archive() do call for the process_source() function that check for key value without checking if key does itself exist, the function call throw an unhandled error exception over the missing key.

** HOW TO PATCH **
The conditional check of the key within the process_source() function should ensure the key does exist AND ensure the conditional value.

We can achieve both using proper dict key check and retrieval using the .get() notation as it either return the dict key value OR None if the key doesn't exist which itself is interpreted as a bool by the conditional check.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/kolla/+/913786

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla/+/913795

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Gaël THEROND <email address hidden>" on branch: stable/zed
Review: https://review.opendev.org/c/openstack/kolla/+/913786
Reason: Create patch from master/MR on master.

[Sven Kieske (s-kieske)]

I'm finally currently trying to reproduce this locally on linux.

Two questions, just to clarify: which version of "kolla" do you use here - I know the code you recognized the culprit also is on master, just to have a complete picture here - master or stable/zed?

How exactly does your "default" kolla-build.conf look like, could you share it - minus private mirror URLs etc.?

Good to see this seems to reproduce on Ubuntu as well, as I would have trouble testing this on Windows.

My test is still running will report back here once I got the results.

[Sven Kieske (s-kieske)]

While trying to reproduce this my shell is hanging during bifrost-deploy:

```
INFO:kolla.common.utils.bifrost-deploy:Built at 2024-03-26 11:27:58.476828 (took 0:00:08.808361)
```

is the last line. I can hit enter and it produces a newline but I get no shell or anything else back.

what I did:

"kolla-build -b ubuntu --openstack-release zed" installed as per the docs via "python3 -m pip install kolla" which installed kolla 17.3.0

Investigating which process is stuck here.

[Sven Kieske (s-kieske)]

so after much fiddling around I must say I currently can not reproduce this. most containers build just fine - except for grafana, horizon and neutron-mlnx-agent, but none provide a stacktrace like it was reported by you.

I guess it will be more clear, once you can provide the information I asked about, because I think my environment might just differ.

Also a minimal reproducer setup (e.g. which containers to build) might be helpful here, as the complete build takes quite some time.

Thanks!

[Maksim Malchuk (mmalchuk)]

this issue not reproduced on Zed because it contains https://review.opendev.org/c/openstack/kolla/+/833644
please check your code or provide more information, e.g. actual kolla-build.conf and detailed logs.

[Gaël THEROND (gtherond)]

Gonna send you everything as soon as possible.

@mmalchuk, the issue lye on the archives for images with plugins, not on the sources themselves, still this code doesn't check the key existing but only its value, which for images with plugins archives sometimes isn't set.

[Maksim Malchuk (mmalchuk)]

ok, thanks, will wait for stuff and will try to reproduce again.