puppet-openstack fuel ci neutron_vlan_ha job fails because environment getting an older version of ostf than on the iso

[Alex Schultz]

The fuel-ci neutron_vlan_ha job for puppet-openstack is currently failing against master because of:

AssertionError: Failed 1 OSTF tests; should fail 0 tests. Names of failed tests:
  - RabbitMQ replication (failure) Failed to establish AMQP connection to 5673/tcp port on 10.109.21.4 from controller node! Please refer to OpenStack logs for more details.

This appears to be related to invalid credentials being used as part of OSTF:
SSHExecCommandFailed: Command 'python -c 'import kombu; c = kombu.Connection("amqp://:@10.109.26.5:5673//"); c.connect()'', exit status: 1, Error:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 237, in connect
    return self.connection
  File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 741, in connection
    self._connection = self._establish_connection()
  File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 696, in _establish_connection
    conn = self.transport.establish_connection()
  File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
    conn = self.Connection(**opts)
  File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 180, in __init__
    (10, 30), # tune
  File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 67, in wait
    self.channel_id, allowed_methods, timeout)
  File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 241, in _wait_method
    channel, method_sig, args, content = read_timeout(timeout)
  File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 330, in read_timeout
    return self.method_reader.read_method()
  File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 189, in read_method
    raise m
IOError: Socket closed

rabbitmq.log:
2016-03-11T14:36:03.435372+00:00 notice: =ERROR REPORT==== 11-Mar-2016::14:35:55 ===
2016-03-11T14:36:03.435372+00:00 notice: Error on AMQP connection <0.23545.1> (10.109.21.4:57117 -> 10.109.21.4:5673, state: starting):
2016-03-11T14:36:03.435377+00:00 notice: AMQPLAIN login refused: user 'guest' - invalid credentials

[Alex Schultz (alex-schultz)]

the job is pulling in an old version of the ostf package. iso 40 has mos915 but the environments are getting mos910

from iso:
[root@nailgun ~]# rpm -qa | grep ostf
fuel-ostf-9.0.0-1.mos915.noarch
[root@nailgun ~]#

from snapshot:
grep ostf centos_installed_rpms.txt
[10.109.25.2] out: fuel-ostf-9.0.0-1.mos910.noarch

[Aleksandra Fedorova (bookwar)]

The issue was caused bu misconfiguration of CI environment: we didn't ckean up old envs and used snapshot of old master node for this test

[Aleksandra Fedorova (bookwar)]

Resolved by cleaning up all ci slaves