(stable/stein) standalone-upgrade job failing tempest post upgrade - test_server_security_groups

[Marios Andreou]

The tripleo-ci-centos-7-standalone-upgrade is failing tempest at [1] - specifically test_list_security_groups_by_server and test_server_security_groups traceback like:

  Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/tempest/api/compute/security_groups/test_security_groups.py", line 149, in test_list_security_groups_by_server
      server_id = self.create_test_server(wait_until='ACTIVE')['id']
    File "/usr/lib/python2.7/site-packages/tempest/api/compute/base.py", line 246, in create_test_server
      **kwargs)
    File "/usr/lib/python2.7/site-packages/tempest/common/compute.py", line 198, in create_test_server
      **kwargs)
    File "/usr/lib/python2.7/site-packages/tempest/lib/services/compute/servers_client.py", line 97, in create_server
      resp, body = self.post('servers', post_body)
    File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 283, in post
      return self.request('POST', url, extra_headers, headers, body, chunked)
    File "/usr/lib/python2.7/site-packages/tempest/lib/services/compute/base_compute_client.py", line 48, in request
      method, url, extra_headers, headers, body, chunked)
    File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 679, in request
      self._error_checker(resp, resp_body)
    File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 780, in _error_checker
      raise exceptions.Forbidden(resp_body, resp=resp)
  tempest.lib.exceptions.Forbidden: Forbidden
  Details: {u'message': u'Only volume-backed servers are allowed for flavors with zero disk.', u'code': 403}

[1] http://logs.openstack.org/67/656967/4/check/tripleo-ci-centos-7-standalone-upgrade/7a25341/logs/tempest.html.gz

[Marios Andreou (marios-b)]

13:12 < chandankumar> marios: kopecmartin arxcruz temprory workaround for tempestconf upgrade https://review.rdoproject.org/r/#/c/20574/

[chandan kumar (chkumar246)]

Above workaroud is not going to work as in upgrade job we run tempest twice and here http://logs.openstack.org/67/656967/4/check/tripleo-ci-centos-7-standalone-upgrade/7a25341/logs/undercloud/var/log/extra/rpm-list.txt.gz -> python2-tempestconf-2.1.1-0.20190424160311.e0d5dd6.el7.noarch is used.

On first run tempest is passing and that time flavor with disk =1 is getting created but on second time, python-tempestconf looks for the flavor by name (remember two flavor with same quote might exists), so it might be tempestconf discovering the flavor with disk < 1
http://logs.openstack.org/67/656967/4/check/tripleo-ci-centos-7-standalone-upgrade/7a25341/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-05-04_22_36_50 which is leading to failure.

It might be a bug in tempestconf as we need to find the flavour differently here making sure it's disk> 0.

[chandan kumar (chkumar246)]

Here is the dnm patch to verify the same: https://review.opendev.org/#/c/657372/

[Martin Kopec (mkopec)]

Facts:
1. python-tempestconf previously did create flavors with disk size = 0 - this got changed by this review [1] only a few months ago
2. [1] is not packaged yet -> python-tempestconf packages for Rocky and Queens don't contain the [1] change
3. flavors with disk size = 0 are forbidden in Stein
4. We're going to update python-tempestconf package for Rocky and Queens in order get there some features implemented over last few months (including [1])
5. When python-tempestconf creates flavors necessary for running Tempest it looks for flavors named 'm1.nano' and 'm1.micro'. If any flavor with such a name is found, python-tempestconf doesn't create a new flavor, it will reuse that one and doesn't matter what are the other specifics of the flavor like disk size , ... -> that said, if there is a flavor named 'm1.nano' or 'm1.micro' with disk size = 0, python-tempestconf will not create a new one, even if it created a flavor with disk size != 0 .. THIS IS NOT A BUG

Now, I don't know which version the job upgrades from, but it doesn't matter. The job is probably failing because when python-tempestconf is executed for the first time (on OpenStack version older that Stein), it creates a flavor with disk size = 0 (because the [1] review is not packaged yet in that version). Then when it is executed again on Stein, it doesn't create a new flavors as the flavors are already created (the fact #5) and that leads to the failure (the reason is the fact #3).

I see two ways how to fix it:
1. make the job delete flavors before the second execution of python-tempestconf
2. update the python-tempestconf package for Rocky and Queens, which we need to do anyway (the fact #4)
I prefer the second way.

[1] https://github.com/openstack/python-tempestconf/commit/72f0edffb0ecb899e278137102915e58b7ddbe23

[Marios Andreou (marios-b)]

@chandan .. do you mean we need to backport that https://review.opendev.org/#/c/637679/ to stable/stein then stable/rocky?

The failing stable-stein job is deploying rocky and upgrade to stein.

If I'm right what is the delay in doing that? Is it not simple to backport?

[Marios Andreou (marios-b)]

apparently chandan has already fixed this, we need temepstconf 2.2.0

13:36 < chandankumar> marios: https://review.rdoproject.org/r/#/c/20579/
13:36 < chandankumar> marios: once promotion happens, new packages will be available

[chandan kumar (chkumar246)]

https://review.rdoproject.org/r/#/c/20579/ -> updates tempestconf version 2.2.0 for stein, rocky and queens will fix the issue.

[Marios Andreou (marios-b)]

related to https://bugs.launchpad.net/tripleo/+bug/1828550 stable/rocky

[Marios Andreou (marios-b)]

current plan. we merge https://review.opendev.org/#/c/655661/7 to land the standalone-upgrade-stein job as voting albeit with skip on those two tests. I just posted https://review.opendev.org/658353 for unskipping once they are green.

---------

apparently it is *rocky* promotion we need, since the fix comes in the source version not the upgraded one and last rocky promotion was 7th may afaics

thanks chandan via irc just now

15:56 < chandankumar> this job upgrades from rocky to stein
15:57 < chandankumar> in rocky we have updated tempestconf-2.2.0 and same in stein also
15:57 < chandankumar> so why lower version is getting installed there?

15:58 < chandankumar> in rocky we donot have promotion na?
15:58 < chandankumar> rocky is blocked due to this https://review.opendev.org/#/c/657964/
15:58 < chandankumar> once rocky got promoted
15:58 < chandankumar> we donot even need to put it in skip list

[Marios Andreou (marios-b)]

rocky promotion on the weekend 11/05 apparently - lets see if that goes green https://review.opendev.org/#/c/658353/ for standalone-upgrade-stein

[Rafael Folco (rafaelfolco)]

can we close this one ?

[chandan kumar (chkumar246)]

New python-temepstconf version is available now

[chandan kumar (chkumar246)]

http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-standalone-upgrade-stein and http://logs.openstack.org/09/661009/3/gate/tripleo-ci-centos-7-standalone-upgrade-stein/d6aa92b/logs/tempest.html.gz tests are passing, closing this bug,