7.0.0.0rc1 Some images failed to build

[Eric Miller]

Just tried to build the latest 7.0.0.0rc1 release on CentOS 7.5 with the latest updates two times and got the same images not building. This is in preparation of using Kolla-Ansible. Seems like having ceilometer-* builds fail might be a major issue?

The command to build the images was:
kolla-build --registry 192.168.254.230:5000 --type source --push

This is the output at the end:

INFO:kolla.common.utils:===========================
INFO:kolla.common.utils:Images that failed to build
INFO:kolla.common.utils:===========================
ERROR:kolla.common.utils:ceilometer-base Failed with status: error
ERROR:kolla.common.utils:ceilometer-central Failed with status: matched
ERROR:kolla.common.utils:ceilometer-compute Failed with status: matched
ERROR:kolla.common.utils:ceilometer-ipmi Failed with status: matched
ERROR:kolla.common.utils:ceilometer-notification Failed with status: matched
ERROR:kolla.common.utils:neutron-server-opendaylight Failed with status: error
ERROR:kolla.common.utils:ovsdpdk-db Failed with status: matched
ERROR:kolla.common.utils:ovsdpdk-vswitchd Failed with status: matched

I include the entire build output in an attachment, but this appears to be where one of the errors occurred:

ERROR:kolla.common.utils.ceilometer-base:Error'd with the following message

ERROR:kolla.common.utils.ceilometer-base:The command '/bin/sh -c ln -s ceilometer-base-source/* ceilometer && pip --no-cache-dir install --upgrade -c /requirements/upper-constraints.txt /ceilometer[gnocchi,mongo,mysql] && mkdir -p /etc/ceilometer && cp -r /ceilometer/etc/ceilometer/* /etc/ceilometer/ && chown -R ceilometer: /etc/ceilometer && sed -i 's|^exec_dirs.*|exec_dirs=/var/lib/kolla/venv/bin,/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin|g' /etc/ceilometer/rootwrap.conf && if [ "$(ls /plugins)" ]; then pip --no-cache-dir install --upgrade -c /requirements/upper-constraints.txt /plugins/*; fi' returned a non-zero code: 1

Eric

[Eduardo Gonzalez (egonzalez90)]

Hi, is an issue in openstack globals requirements. A patch is under review, once merged build will succeed. Affect both master and rocky branch.

Regards

[Eric Miller (erickmiller)]

Thanks Eduardo! Due to ignorance of the process, can you help me understand how something like this gets past all of the Ci/CD build processes? Again, I don't know much about this process, but this seems like a pretty significant issue that should have been caught before it affected everyone from deploying OpenStack (at least with Kolla)?

Eric

[Eduardo Gonzalez (egonzalez90)]

Hi, this wasn't catched at CI because was a change outside kolla and after rocky branch was created.
A pip package was created for ceilometer which wasn't before, this affect any project installing ceilometer.

In any case, rocky is not released in kolla yet, is only a release candidate.

https://review.openstack.org/#/c/601487/
Regards

[Eduardo Gonzalez (egonzalez90)]

Related parts of the issue:

networking-odl installing ceilometer from git instead of a pip package https://github.com/openstack/networking-odl/blob/master/requirements.txt#L21
ceilometer recently packaged and published of pypi https://pypi.org/project/ceilometer/
Ceilometer not being pàrt of openstack requirements https://review.openstack.org/#/c/601487/
We installing every pip package validating against openstack requirements https://github.com/openstack/kolla/blob/master/docker/macros.j2#L27

All this together causes build failures if trying to install ceilometer (even fron git) or as a requirement of other packages.

If using binary images there is not issue in the build as other deployment tools does not have such issue. Only the ones which uses source code and rely on the global requirements.

Regards

[Eric Miller (erickmiller)]

Thanks for the explanation! I'm still learning, so this helps put the pieces together.

Btw, I have been using the binary images since the source builds have failed. Unfortunately, there are a few binary images missing, that are included when using the source build, such as monasca. Do you know why there is a discrepancy?

Eric

[Eduardo Gonzalez (egonzalez90)]

Probably there is no binary available for the distro used.

[Eric Miller (erickmiller)]

Aren't the binaries just builds using the source options and pushed to the Docker repository? I don't know when/where this happens, so maybe I'm missing a piece as to why some binaries are created for CentOS (for example) and others are not.

[Eduardo Gonzalez (egonzalez90)]

Binaries are the rpms or deb files created by Ubuntu and RDO. Sources are the installation of pip packages directly without any modifications or downstream work specific to a distro.

[Eduardo Gonzalez (egonzalez90)]

Having a binary Package depends on the work of external project, while the pip packages are created during openstack releases

[Eric Miller (erickmiller)]

Hi Eduardo,

If I understood the review page you mentioned:
https://review.openstack.org/#/c/601487/

seems to indicate that the patch was merged?
https://review.openstack.org/#/q/1d44f821536a970cd804abe31cba3bcd5188bcb1

It looks like the kolla-build from source still fails to build ceilometer images, though - so maybe there is another piece that needs to be merged that I'm not aware of. Again, just trying to understand the process so I can track the progress.

Not having Ceilometer to test with is quite a problem. :)

Thanks for your help!

Eric

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/605351

[Mark Goddard (mgoddard)]

Eric, I've proposed a change [1] to revert Eduardo's change [2] in kolla that removed ceilometer from upper-constraints, which should work now that [3] has merged. However, it is my understanding that ceilometer builds were fixed by that workaround.

[1] https://review.openstack.org/605351
[2] https://review.openstack.org/603357
[3] https://review.openstack.org/601487

[Eric Miller (erickmiller)]

Thanks Mark! I had uninstalled, re-installed, and run kolla-build with the source option very recent and the problem still remained. However, it looks like the changes Eduardo made (and you reverted) were to master? Does that mean these changes were backported to the kolla 7.0.0.0rc1 build? or only available in master? Maybe that's my problem - using 7.0.0.0rc?

I have been using:
pip install kolla==$kolla_version

where $kolla_version=7.0.0.0rc1

Eric

[Mark Goddard (mgoddard)]

Yes, Eduardo's change was merged after 7.0.0.0rc1. I'd recommend trying the
stable/rocky branch.

On Wed, 26 Sep 2018 at 09:45, Eric Miller <email address hidden>
wrote:

> Thanks Mark! I had uninstalled, re-installed, and run kolla-build with
> the source option very recent and the problem still remained. However,
> it looks like the changes Eduardo made (and you reverted) were to
> master? Does that mean these changes were backported to the kolla
> 7.0.0.0rc1 build? or only available in master? Maybe that's my problem
> - using 7.0.0.0rc?
>
> I have been using:
> pip install kolla==$kolla_version
>
> where $kolla_version=7.0.0.0rc1
>
> Eric
>
> --
> You received this bug notification because you are a member of kolla-
> drivers, which is subscribed to kolla.
> Matching subscriptions: kolla-bug
> https://bugs.launchpad.net/bugs/1792270
>
> Title:
> 7.0.0.0rc1 Some images failed to build
>
> Status in kolla:
> New
>
> Bug description:
> Just tried to build the latest 7.0.0.0rc1 release on CentOS 7.5 with
> the latest updates two times and got the same images not building.
> This is in preparation of using Kolla-Ansible. Seems like having
> ceilometer-* builds fail might be a major issue?
>
> The command to build the images was:
> kolla-build --registry 192.168.254.230:5000 --type source --push
>
> This is the output at the end:
>
> INFO:kolla.common.utils:===========================
> INFO:kolla.common.utils:Images that failed to build
> INFO:kolla.common.utils:===========================
> ERROR:kolla.common.utils:ceilometer-base Failed with status: error
> ERROR:kolla.common.utils:ceilometer-central Failed with status: matched
> ERROR:kolla.common.utils:ceilometer-compute Failed with status: matched
> ERROR:kolla.common.utils:ceilometer-ipmi Failed with status: matched
> ERROR:kolla.common.utils:ceilometer-notification Failed with status:
> matched
> ERROR:kolla.common.utils:neutron-server-opendaylight Failed with status:
> error
> ERROR:kolla.common.utils:ovsdpdk-db Failed with status: matched
> ERROR:kolla.common.utils:ovsdpdk-vswitchd Failed with status: matched
>
>
> I include the entire build output in an attachment, but this appears to
> be where one of the errors occurred:
>
> ERROR:kolla.common.utils.ceilometer-base:Error'd with the following
> message
>
> ERROR:kolla.common.utils.ceilometer-base:The command '/bin/sh -c ln -s
> ceilometer-base-source/* ceilometer && pip --no-cache-dir install
> --upgrade -c /requirements/upper-constraints.txt
> /ceilometer[gnocchi,mongo,mysql] && mkdir -p /etc/ceilometer
> && cp -r /ceilometer/etc/ceilometer/* /etc/ceilometer/ && chown -R
> ceilometer: /etc/ceilometer && sed -i
>
> 's|^exec_dirs.*|exec_dirs=/var/lib/kolla/venv/bin,/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin|g'
> /etc/ceilometer/rootwrap.conf && if [ "$(ls /plugins)" ]; then
> pip --no-cache-dir install --upgrade -c /requirements/upper-
> constraints.txt /plugins/*; fi' returned a non-zero code: 1
>
>
> Eric
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/kolla/+bug/1792270/+subscriptio...

[Eric Miller (erickmiller)]

It doesn't look like Kolla has a stable/rocky version?
https://releases.openstack.org/teams/kolla.html

[Mark Goddard (mgoddard)]

It's not a release, it's a git branch. You can clone it from here:
https://github.com/openstack/kolla/tree/stable/rocky

On Wed, 26 Sep 2018 at 10:15, Eric Miller <email address hidden>
wrote:

> It doesn't look like Kolla has a stable/rocky version?
> https://releases.openstack.org/teams/kolla.html
>
> --
> You received this bug notification because you are a member of kolla-
> drivers, which is subscribed to kolla.
> Matching subscriptions: kolla-bug
> https://bugs.launchpad.net/bugs/1792270
>
> Title:
> 7.0.0.0rc1 Some images failed to build
>
> Status in kolla:
> New
>
> Bug description:
> Just tried to build the latest 7.0.0.0rc1 release on CentOS 7.5 with
> the latest updates two times and got the same images not building.
> This is in preparation of using Kolla-Ansible. Seems like having
> ceilometer-* builds fail might be a major issue?
>
> The command to build the images was:
> kolla-build --registry 192.168.254.230:5000 --type source --push
>
> This is the output at the end:
>
> INFO:kolla.common.utils:===========================
> INFO:kolla.common.utils:Images that failed to build
> INFO:kolla.common.utils:===========================
> ERROR:kolla.common.utils:ceilometer-base Failed with status: error
> ERROR:kolla.common.utils:ceilometer-central Failed with status: matched
> ERROR:kolla.common.utils:ceilometer-compute Failed with status: matched
> ERROR:kolla.common.utils:ceilometer-ipmi Failed with status: matched
> ERROR:kolla.common.utils:ceilometer-notification Failed with status:
> matched
> ERROR:kolla.common.utils:neutron-server-opendaylight Failed with status:
> error
> ERROR:kolla.common.utils:ovsdpdk-db Failed with status: matched
> ERROR:kolla.common.utils:ovsdpdk-vswitchd Failed with status: matched
>
>
> I include the entire build output in an attachment, but this appears to
> be where one of the errors occurred:
>
> ERROR:kolla.common.utils.ceilometer-base:Error'd with the following
> message
>
> ERROR:kolla.common.utils.ceilometer-base:The command '/bin/sh -c ln -s
> ceilometer-base-source/* ceilometer && pip --no-cache-dir install
> --upgrade -c /requirements/upper-constraints.txt
> /ceilometer[gnocchi,mongo,mysql] && mkdir -p /etc/ceilometer
> && cp -r /ceilometer/etc/ceilometer/* /etc/ceilometer/ && chown -R
> ceilometer: /etc/ceilometer && sed -i
>
> 's|^exec_dirs.*|exec_dirs=/var/lib/kolla/venv/bin,/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin|g'
> /etc/ceilometer/rootwrap.conf && if [ "$(ls /plugins)" ]; then
> pip --no-cache-dir install --upgrade -c /requirements/upper-
> constraints.txt /plugins/*; fi' returned a non-zero code: 1
>
>
> Eric
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/kolla/+bug/1792270/+subscriptions
>

[Eric Miller (erickmiller)]

Thanks Mark! I learned a bit more about packages and how to install the development version from git (using setup.py) and was able to get the latest stable/rocky build installed (Kolla and Kolla-Ansible).

The Ceilometer Kolla container build issue is fixed, of course. :)

Thank you!

Eric

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mark Goddard (<email address hidden>) on branch: master
Review: https://review.openstack.org/605351