Network verification failed: Queue.declare: (406) PRECONDITION_FAILED

[Artem Panchenko]

{"build_id": "2014-04-29_01-00-26", "mirantis": "yes", "build_number": "155", "ostf_sha": "134765fcb5a07dce0cd1bb399b2290c988c3c63b", "nailgun_sha": "43fdef4393e64a7c23df844a11ad3f8fc820283b", "production": "docker", "api": "1.0", "fuelmain_sha": "b33adfc3a17377415bf0fc80b7b8419fc5fb7176", "astute_sha": "3cffebde1e5452f5dbf8f744c6525fc36c7afbf3", "release": "5.0", "fuellib_sha": "68b11620e7b0d42eafde375ef4992cadc8b0d10e"}

Steps to reproduce:

1. Create new environment:

 - Ubuntu
 - Multinode + HA
 - Neutron + Vlan

2. Add 3 controllers, 1 compute and 1 cinder

3. Go to "Networks" tab, edit settings, save changes and click the "Vetify Networks" button. It returns error:

Verification failed.
Queue.declare: (406) PRECONDITION_FAILED - parameters for queue 'naily' in vhost '/' not equivalent

Here is debug log:

root@fuelweb ~]# fuel --debug network --verify --env 1
GET http://127.0.0.1:8000/api/v1/clusters/1/
GET http://127.0.0.1:8000/api/v1/clusters/1/network_configuration/neutron
PUT http://127.0.0.1:8000/api/v1/clusters/1/network_configuration/neutron/verify data={"networking_parameters": {"internal_gateway": "192.168.111.1", "dns_nameservers": ["8.8.4.4", "8.8.8.8"], "floating_ranges": [["172.18.166.225", "172.18.166.254"]], "net_l23_provider": "ovs", "base_mac": "fa:16:3e:00:00:00", "internal_cidr": "192.168.111.0/24", "gre_id_range": [2, 65535], "vlan_range": [1000, 1030], "segmentation_type": "vlan"}, "management_vip": "192.168.0.2", "networks": [{"name": "public", "ip_ranges": [["172.18.166.198", "172.18.166.224"]], "gateway": "172.18.166.193", "meta": {"name": "public", "notation": "ip_ranges", "render_type": null, "assign_vip": true, "map_priority": 1, "floating_range_var": "floating_ranges", "use_gateway": true, "vlan_start": null, "render_addr_mask": "public", "cidr": "172.16.0.0/24", "configurable": true, "ip_range": ["172.16.0.2", "172.16.0.126"]}, "cluster_id": 1, "vlan_start": 378, "cidr": "172.18.166.192/26", "id": 2}, {"name": "management", "ip_ranges": [["192.168.0.2", "192.168.0.254"]], "gateway": null, "meta": {"name": "management", "notation": "cidr", "render_type": "cidr", "assign_vip": true, "map_priority": 2, "use_gateway": false, "vlan_start": 101, "render_addr_mask": "internal", "cidr": "192.168.0.0/24", "configurable": true}, "cluster_id": 1, "vlan_start": 101, "cidr": "192.168.0.0/24", "id": 3}, {"name": "storage", "ip_ranges": [["192.168.1.2", "192.168.1.254"]], "gateway": null, "meta": {"name": "storage", "notation": "cidr", "render_type": "cidr", "assign_vip": false, "map_priority": 2, "use_gateway": false, "vlan_start": 102, "render_addr_mask": "storage", "cidr": "192.168.1.0/24", "configurable": true}, "cluster_id": 1, "vlan_start": 102, "cidr": "192.168.1.0/24", "id": 4}, {"name": "private", "ip_ranges": [], "gateway": null, "meta": {"name": "private", "notation": null, "render_type": null, "assign_vip": false, "map_priority": 2, "use_gateway": false, "vlan_start": null, "render_addr_mask": null, "neutron_vlan_range": true, "configurable": false, "seg_type": "vlan"}, "cluster_id": 1, "vlan_start": null, "cidr": null, "id": 5}, {"name": "fuelweb_admin", "ip_ranges": [["10.20.204.3", "10.20.204.127"]], "gateway": null, "meta": {"notation": "cidr", "render_type": null, "assign_vip": false, "configurable": false, "unmovable": true, "use_gateway": false, "render_addr_mask": null, "map_priority": 0}, "cluster_id": null, "vlan_start": null, "cidr": "10.20.204.0/24", "id": 1}], "public_vip": "172.18.166.198"}
Verification status is 'error'. message: Queue.declare: (406) PRECONDITION_FAILED - parameters for queue 'naily' in vhost '/' not equivalent

Also you can find docker-nailgun.log in attachments.

[Artem Panchenko (apanchenko-8)]

Cluster deployment also failed due to AMQP error:

http://paste.openstack.org/show/77672/

Archive of rabbitmq logs is attached.

[Artem Panchenko (apanchenko-8)]

Can't reproduce this issue on ISO # 170, changing bug status to incomplete

[Mike Scherbakov (mihgen)]

Thanks for reporting. It was likely to occur due to containers startup issues. Closing as Invalid for now, please reopen and notify if you face this ever again.

[Andrey Sledzinskiy (asledzinskiy)]

We faced this issue in system tests - got the same error when generating diagnostic snapshot

Error Message

nonnumeric port: ' (406) PRECONDITION_FAILED - parameters for queue 'naily' in vhost ''

Stacktrace

Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 332, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1044, in runTest
    self._testFunc()
  File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/5.0_fuelmain.system_test.centos.thread_3/fuelweb_test/helpers/decorators.py", line 56, in wrapper
    create_diagnostic_snapshot(args[0].env, "fail", func.__name__)
  File "/home/jenkins/workspace/5.0_fuelmain.system_test.centos.thread_3/fuelweb_test/helpers/decorators.py", line 156, in create_diagnostic_snapshot
    save_logs(url, os.path.join(settings.LOGS_DIR, log_file_name))
  File "/home/jenkins/workspace/5.0_fuelmain.system_test.centos.thread_3/fuelweb_test/helpers/decorators.py", line 34, in save_logs
    urllib2.urlopen(url).read()
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1153, in do_open
    h = http_class(host, timeout=req.timeout) # will parse host:port
  File "/usr/lib/python2.7/httplib.py", line 704, in __init__
    self._set_hostport(host, port)
  File "/usr/lib/python2.7/httplib.py", line 732, in _set_hostport
    raise InvalidURL("nonnumeric port: '%s'" % host[i+1:])
InvalidURL: nonnumeric port: ' (406) PRECONDITION_FAILED - parameters for queue 'naily' in vhost ''

Logs are attached