BVT_2 for custom iso failed with error: Non-empty feature list specified without CPU model

[Sergey Kulanov]

BVT [1] test failed with error:

<<<<<##############################[ prepare_release ]##############################>>>>>
Prepare master node

        Scenario:
            1. Revert snapshot "empty"
            2. Download the release if needed. Uploads custom manifest.

        Snapshot: ready

2016-01-28 08:36:38,274 - INFO base_test_case.py:114 --
                                                       <<< 1. Revert snapshot "empty" >>>
2016-01-28 08:36:38,294 - INFO environment.py:355 -- We have snapshot with such name: empty
2016-01-28 08:36:38,294 - INFO environment.py:357 -- Reverting the snapshot 'empty' ....
libvirt: CPU Driver error : XML error: Non-empty feature list specified without CPU model
libvirt: CPU Driver error : XML error: Non-empty feature list specified without CPU model
libvirt: CPU Driver error : XML error: Non-empty feature list specified without CPU model
libvirt: CPU Driver error : XML error: Non-empty feature list specified without CPU model
libvirt: CPU Driver error : XML error: Non-empty feature list specified without CPU model
libvirt: CPU Driver error : XML error: Non-empty feature list specified without CPU model
libvirt: CPU Driver error : XML error: Non-empty feature list specified without CPU model
libvirt: CPU Driver error : XML error: Non-empty feature list specified without CPU model
libvirt: CPU Driver error : XML error: Non-empty feature list specified without CPU model
libvirt: CPU Driver error : XML error: Non-empty feature list specified without CPU model
2016-01-28 08:37:54,580 - ERROR __init__.py:67 -- generate_logs raised: URLError(error(113, 'No route to host'),)
Traceback: Traceback (most recent call last):
  File "/home/jenkins/workspace/9.0.fuel_community.ubuntu.bvt_2/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.fuel_community.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 135, in wrapped
    response = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.fuel_community.ubuntu.bvt_2/fuelweb_test/models/nailgun_client.py", line 355, in generate_logs
    return self.client.put("/api/logs/package")
  File "/home/jenkins/workspace/9.0.fuel_community.ubuntu.bvt_2/fuelweb_test/helpers/http.py", line 83, in put
    return self._open(req)
  File "/home/jenkins/workspace/9.0.fuel_community.ubuntu.bvt_2/fuelweb_test/helpers/http.py", line 92, in _open
    return self._get_response(req)
  File "/home/jenkins/workspace/9.0.fuel_community.ubuntu.bvt_2/fuelweb_test/helpers/http.py", line 116, in _get_response
    return self.opener.open(req)
  File "/usr/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 113] No route to host>

[1]. https://ci.fuel-infra.org/job/9.0.fuel_community.ubuntu.bvt_2/

[Sergey Kulanov (skulanov)]

btw, the same issue for the rest bvt: https://ci.fuel-infra.org/view/ISO/job/7.0.fuel_community.ubuntu.bvt_2/

[Sergey Kulanov (skulanov)]

compared libvirt against product-ci:

Product-ci:
ii libvirt-bin 1.2.2-0ubuntu13.1.14 amd64 programs for the libvirt library
ii libvirt-dev 1.2.2-0ubuntu13.1.14 amd64 development files for the libvirt library
ii libvirt0 1.2.2-0ubuntu13.1.14 amd64 library for interfacing with different virtualization systems
ii python-libvirt 1.2.2-0ubuntu2 amd64 libvirt Python bindings
Fuel-ci:
ii libvirt-bin 1.2.2-0ubuntu13.1.9 amd64 programs for the libvirt library
ii libvirt-dev 1.2.2-0ubuntu13.1.9 amd64 development files for the libvirt library
ii libvirt0 1.2.2-0ubuntu13.1.9 amd64 library for interfacing with different virtualization systems
ii python-libvirt 1.2.2-0ubuntu2 amd64 libvirt Python bindings

[Dmitry Kaigarodеsev (dkaiharodsev)]

@fuel-devops : please update libvirt deb package up to 1.2.2-0ubuntu13.1.14

[Sergey Kulanov (skulanov)]

I'd propose to use the latest one:

apt-cache policy libvirt-bin
libvirt-bin:
  Installed: 1.2.2-0ubuntu13.1.16
  Candidate: 1.2.2-0ubuntu13.1.16
  Version table:
 *** 1.2.2-0ubuntu13.1.16 0
        500 http://ua.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
        100 /var/lib/dpkg/status
     1.2.2-0ubuntu13 0
        500 http://ua.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

[Igor Shishkin (teran)]

Looks like here's a root cause: https://github.com/openstack/fuel-devops/commit/b25c7c7ffa7fe25055333c1ca701dcf245324daa

My suggestion is to revert first and upgrade nodes in normall flow instead of like we doing it now: ASAP mode :(

[Igor Shishkin (teran)]

PS. fuel-devops downgrade is faster than upgrade for libvirt since package upgrade requires pretty huge job on nodes manipulation.

[Nastya Urlapova (aurlapova)]

@Igor, for d93a3a7 fuel-qa we use 2.9.16 fuel devops https://github.com/openstack/fuel-qa/blob/master/fuelweb_test/requirements.txt#L2
why did you use fuel-devops version 2.9.17 is question for you.

[Igor Shishkin (teran)]

@Nastya, please assign bugs to the team who is responsible for that: CI team in such case.

[Dmitry Kaigarodеsev (dkaiharodsev)]

I can see from the latest 'prepare_env' job that we're using 2.9.16 on related slaves:

+ [[ -n '' ]]
+ curl -fsS https://raw.githubusercontent.com/openstack/fuel-qa/master/fuelweb_test/requirements.txt
+ pip install -r /home/jenkins/workspace/prepare_env/venv-requirements.txt --upgrade
Collecting git+git://github.com/openstack/fuel-devops.git@2.9.16 (from -r /home/jenkins/workspace/prepare_env/venv-requirements.txt (line 2))
  Cloning git://github.com/openstack/fuel-devops.git (to 2.9.16) to /tmp/pip-XZsZfj-build

[Igor Shishkin (teran)]

Btw, LOL

(venv-nailgun-tests-2.9)jenkins@ci-slave04:~/venv-nailgun-tests-2.9$ pip freeze | fgrep dev
devops==2.9.9
fuel-devops==2.9.16
rst2pdf==0.93.dev0

https://ci.fuel-infra.org/job/9.0.fuel_community.ubuntu.bvt_2/20/console

[Igor Shishkin (teran)]

CI-team, please recreate venvs with python-devops and ensure we have the only version of the lib.

[Dmitry Kaigarodеsev (dkaiharodsev)]

we've avoid this issue by using fresh deployed slave

[Roman Podoliaka (rpodolyaka)]

Looks like this is reproduced again:

https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/1388/
https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/1380/
https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/1378/

^ all 3 failures happened on the same CI node - ci-slave02.fuel-infra.org

[Dmitry Kaigarodеsev (dkaiharodsev)]

related slave has been disabled until fix and recheck

[Roman Vyalov (r0mikiam)]

Slave 02 was disabled, now bvt tests are green