Test deploy_ceph_net_tmpl has failed with error "Some services are listening on wrong IPs!"

[Ilya Bumarskov]

Fuel 10 build #1501

Scenario:
            1. Revert snapshot with 5 slaves
            2. Create cluster (HA) with Neutron VLAN/VXLAN/GRE
            3. Add 3 controller + ceph nodes
            4. Add 2 compute + ceph nodes
            5. Upload 'ceph' network template
            6. Create custom network groups basing
               on template endpoints assignments
            7. Run network verification
            8. Deploy cluster
            9. Run network verification
            10. Run health checks (OSTF)
            11. Check L3 network configuration on slaves
            12. Check that services are listening on their networks only

Actual result:
Traceback (most recent call last):
  File "/home/jenkins/workspace/10.0.system_test.ubuntu.network_templates/core/helpers/log_helpers.py", line 204, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0.system_test.ubuntu.network_templates/fuelweb_test/tests/test_net_templates_base.py", line 441, in check_services_networks
    'Some services are listening on wrong IPs! '
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 163, in assert_true
    raise ASSERTION_ERROR(message)
AssertionError: Some services are listening on wrong IPs! Please check logs for details!

[Ilya Bumarskov (ibumarskov)]

Snapshots can be found by link: https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.network_templates/216/

[Ilya Bumarskov (ibumarskov)]

Firstly necessary to investigate where is problem (Test or Prod), so I have assigned it on QA team.

[Vladimir Khlyunev (vkhlyunev)]

2017-03-24 03:33:21 - ERROR test_net_templates_base.py:421 -- Service "ceph-radosgw" (port 6780/tcp) is listening on wrong IP address(es) on "node-1": expected "set(['10.200.254.1'])", got "set(['0.0.0.0'])"!
2017-03-24 03:33:25 - ERROR test_net_templates_base.py:421 -- Service "ceph-radosgw" (port 6780/tcp) is listening on wrong IP address(es) on "node-3": expected "set(['10.200.254.3'])", got "set(['0.0.0.0'])"!
2017-03-24 03:33:27 - ERROR test_net_templates_base.py:421 -- Service "ceph-radosgw" (port 6780/tcp) is listening on wrong IP address(es) on "node-2": expected "set(['10.200.254.2'])", got "set(['0.0.0.0'])"!

dev-team, smth changed in rados config; please investigate it

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/450638

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/450638
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=152f029f38d53382b54ae4e0f2ee58ec48530fef
Submitter: Jenkins
Branch: master

commit 152f029f38d53382b54ae4e0f2ee58ec48530fef
Author: Michael Polenchuk <email address hidden>
Date: Tue Mar 28 12:40:03 2017 +0400

    [ceph] Change radosgw frontend port

    Set radosgw listen port to default
    value of built-in frontend.

    Change-Id: I4ddcba69018238001df9b00b9e4174de3a9840c9
    Related-Bug: #1674917

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: master
Review: https://review.opendev.org/450632
Reason: This repo is retired now, no further work will get merged.