AssertionError: Task 'deploy' has incorrect status. error != ready

[Ann Taraday]

Reproduced on https://product-ci.infra.mirantis.net/view/10.0/job/10.0.main.ubuntu.bvt_2/527/console

Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/10.0.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 117, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0.main.ubuntu.bvt_2/fuelweb_test/tests/test_ceph.py", line 505, in ceph_rados_gw
    self.fuel_web.deploy_cluster_wait(cluster_id)
  File "/home/jenkins/workspace/10.0.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 459, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 444, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 495, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 502, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 386, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 931, in deploy_cluster_wait
    self.assert_task_success(task, interval=interval, timeout=timeout)
  File "/home/jenkins/workspace/10.0.main.ubuntu.bvt_2/fuelweb_test/__init__.py", line 60, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 333, in assert_task_success
    task["name"], task['status'], 'ready', _message(task)
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Provision has failed. Failed to execute hook 'shell' Failed to run command cd / && fa_build_image --image_build_dir /var/lib/fuel/ibp --log-file /var/log/fuel-agent-env-1.log --data_driver nailgun_build_image --input_data '{"image_data": {"/boot": {"container": "gzip", "uri": "http://10.109.0.2:8080/targetimages/env_1_ubuntu_1604_amd64-boot.img.gz", "format": "ext2"}, "/": {"container": "gzip", "uri": "http://10.109.0.2:8080/targetimages/env_1_ubuntu_1604_amd64.img.gz", "format": "ext4"}}, "output": "/var/www/nailgun/targetimages", "repos": [{"name": "ubuntu-0", "section": "main universe multiverse", "uri": "http://cz.archive.ubuntu.com/ubuntu/", "priority": 1001, "suite": "xenial", "type": "deb"}, {"name": "ubuntu-1", "section": "main universe multiverse", "uri": "http://cz.archive.ubuntu.com/ubuntu/", "priority": 1001, "suite": "xenial-updates", "type": "deb"}, {"name": "ubuntu-2", "section": "main universe multiverse", "uri": "http://cz.archive.ubuntu.com/ubuntu/", "priority": 1001, "suite": "xenial-security", "type": "deb"}, {"name": "uca", "section": "main", "uri": "http://ubuntu-cloud.archive.canonical.com/ubuntu", "priority": 1080, "suite": "xenial-updates/newton", "type": "deb"}, {"name": "uca-proposed", "section": "main", "uri": "http://ubuntu-cloud.archive.canonical.com/ubuntu", "priority": 1080, "suite": "xenial-proposed/newton", "type": "deb"}, {"name": "mos", "section": "main restricted", "uri": "http://10.109.0.2:8080/newton-10.0/ubuntu/x86_64", "priority": 1050, "suite": "mos-master", "type": "deb"}, {"name": "mos-updates", "section": "main restricted", "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/10.0/", "priority": 1050, "suite": "mos10.0-updates", "type": "deb"}, {"name": "mos-security", "section": "main restricted", "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/10.0/", "priority": 1050, "suite": "mos10.0-security", "type": "deb"}, {"name": "mos-holdback", "section": "main restricted", "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/10.0/", "priority": 1100, "suite": "mos10.0-holdback", "type": "deb"}, {"name": "Auxiliary", "section": "main restricted", "uri": "http://10.109.0.2:8080/newton-10.0/ubuntu/auxiliary", "priority": 1150, "suite": "auxiliary", "type": "deb"}], "packages": ["acl", "anacron", "bash-completion", "bridge-utils", "bsdmainutils", "build-essential", "cloud-init", "curl", "daemonize", "debconf-utils", "gdisk", "grub-pc", "hwloc", "linux-firmware", "linux-headers-generic-lts-xenial", "linux-image-generic-lts-xenial", "lvm2", "mcollective", "mdadm", "multipath-tools", "multipath-tools-boot", "nailgun-agent", "nailgun-mcagents", "network-checker", "ntp", "ntpdate", "openssh-client", "openssh-server", "puppet", "python-amqp", "ruby-augeas", "ruby-ipaddress", "ruby-json", "ruby-netaddr", "ruby-openstack", "ruby-shadow", "ruby-stomp", "telnet", "ubuntu-minimal", "ubuntu-standard", "uuid-runtime", "vim", "virt-what", "vlan"], "codename": "xenial"}' (node master returned 255).'

[Oleksiy Molchanov (omolchanov)]

Command: chroot /var/lib/fuel/ibp/tmp0EYWiN.fuel-agent-image apt-get -y install acl anacron bash-completion bridge-utils bsdmainutils build-essential cloud-init curl daemonize debconf-utils gdisk grub-pc hwloc linux-firmware linux-headers-generic-lts-xenial linux-image-generic-lts-xenial lvm2 mcollective mdadm multipath-tools multipath-tools-boot nailgun-agent nailgun-mcagents network-checker ntp ntpdate openssh-client openssh-server puppet python-amqp ruby-augeas ruby-ipaddress ruby-json ruby-netaddr ruby-openstack ruby-shadow ruby-stomp telnet ubuntu-minimal ubuntu-standard uuid-runtime vim virt-what vlan
Exit code: 100
Stdout: 'Reading package lists...\nBuilding dependency tree...\nReading state information...\nubuntu-minimal is already the newest version (1.361).\nSome packages could not be installed. This may mean that you have\nrequested an impossible situation or if you are using the unstable\ndistribution that some required packages have not yet been created\nor been moved out of Incoming.\nThe following information may help to resolve the situation:\n\nThe following packages have unmet dependencies:\n network-checker : Depends: python-requests (>= 2.10.0) but 2.9.1-3 is to be installed\n'
Stderr: 'E: Unable to correct problems, you have held broken packages.\n'

[Dmitry Teselkin (teselkin-d)]

https://review.openstack.org/#/c/352922/

[Dmitry Teselkin (teselkin-d)]

We've never saw this issue on 9.x