OSTF Failed: Can not assign floating ip to instance

[Ivan Kolodyazhny]

2016-12-13T13:59:26.939801+00:00 err: 2016-12-13 13:59:26.929 27890 ERROR nova.api.openstack.compute.floating_ips [req-8d5ab008-5cbf-410a-9c5e-1d291fc3e616 b
8b9158525d94f60800636f09322cadd df18f2f208594011a26192ebc4d5af1c - - -] Unable to associate floating IP 10.109.18.130 to fixed IP 10.109.19.7 for instance 6a3
4deef-7b3e-4d24-94c7-88c158e9f77e. Error: External network 4494d0b3-b907-49ff-a11f-8dadf0a9a0f7 is not reachable from subnet 08268648-31b4-4631-ac42-ba17d171a
e8e. Therefore, cannot associate Port 13e38517-a8c4-412b-8b7d-dc5300e11ec5 with a Floating IP.
Neutron server returns request_ids: ['req-27375158-2980-4c45-9e7f-c4212f1d7561']
2016-12-13 13:59:26.929 27890 ERROR nova.api.openstack.compute.floating_ips Traceback (most recent call last):
2016-12-13 13:59:26.929 27890 ERROR nova.api.openstack.compute.floating_ips File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/floating_ips.py", line 240, in _add_floating_ip
2016-12-13 13:59:26.929 27890 ERROR nova.api.openstack.compute.floating_ips fixed_address=fixed_address)
2016-12-13 13:59:26.929 27890 ERROR nova.api.openstack.compute.floating_ips File "/usr/lib/python2.7/dist-packages/nova/network/base_api.py", line 77, in wrapper
2016-12-13 13:59:26.929 27890 ERROR nova.api.openstack.compute.floating_ips res = f(self, context, *args, **kwargs)
2016-12-13 13:59:26.929 27890 ERROR nova.api.openstack.compute.floating_ips File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1268, in associate_floating_ip
2016-12-13 13:59:26.929 27890 ERROR nova.api.openstack.compute.floating_ips client.update_floatingip(fip['id'], {'floatingip': param})
2016-12-13 13:59:26.929 27890 ERROR nova.api.openstack.compute.floating_ips File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 97, in with_params
2016-12-13 13:59:26.929 27890 ERROR nova.api.openstack.compute.floating_ips ret = self.function(instance, *args, **kwargs)
2016-12-13 13:59:26.929 27890 ERROR nova.api.openstack.compute.floating_ips F
2016-12-13T13:59:26.940775+00:00 info: 2016-12-13 13:59:26.939 27890 INFO nova.api.openstack.wsgi [req-8d5ab008-5cbf-410a-9c5e-1d291fc3e616 b8b9158525d94f60800636f09322cadd df18f2f208594011a26192ebc4d5af1c - - -] HTTP exception thrown: Unable to associate floating IP 10.109.18.130 to fixed IP 10.109.19.7 for instance 6a34deef-7b3e-4d24-94c7-88c158e9f77e. Error: External network 4494d0b3-b907-49ff-a11f-8dadf0a9a0f7 is not reachable from subnet 08268648-31b4-4631-ac42-ba17d171ae8e. Therefore, cannot associate Port 13e38517-a8c4-412b-8b7d-dc5300e11ec5 with a Floating IP.

[Ivan Kolodyazhny (e0ne)]

From Roman P.:

I'm not sure that external_gateway_info should be empty
root@node-1:~# neutron router-list
+--------------------------------------+----------+-----------------------+-------------+-------+
| id | name | external_gateway_info | distributed | ha |
+--------------------------------------+----------+-----------------------+-------------+-------+
| a94b10ab-996f-4916-9b2f-ff11011fff4e | router04 | null | False | False |
+--------------------------------------+----------+-----------------------+-------------+-------+

[Dmitry Teselkin (teselkin-d)]

https://product-ci.infra.mirantis.net/job/9.x.snapshot/629/

[Denis Meltsaykin (dmeltsaykin)]

Nikita, could you please take a look on that?

[Nikita Karpin (mkarpin)]

ok, looking

[Nikita Karpin (mkarpin)]

it is reproduced here too for example - https://ci.fuel-infra.org/job/master.fuel-library.pkgs.ubuntu.neutron_vlan_ha/12823/

[Nikita Karpin (mkarpin)]

I found that there was a split-brain issue in cluster during deploy -

node-1-10.109.36.4/var/log/daemon.log:<27>Dec 14 14:40:04 node-1 ocf-mysql-wss: ERROR: p_mysqld: check_if_galera_pc(): But I'm running a new cluster, PID:9896, this is a split-brain!
node-1-10.109.36.4/var/log/syslog:<27>Dec 14 14:40:04 node-1 ocf-mysql-wss: ERROR: p_mysqld: check_if_galera_pc(): But I'm running a new cluster, PID:9896, this is a split-brain!

But I am not sure its related, will check puppet tasks ordering too

[Nikita Karpin (mkarpin)]

Looks like i found the reason:

Here [0] we have 2 openstack-network-routers task which apply the same manifests but twice - once on openstack-network configuration stage and the other on post deployment stage.

I reverted the environment with the bug and found that openstack_network/routers.pp was applied only one time. In puppet logs on primary controller I see that it was applied only once:

grep openstack_network/routers /var/log/puppet.log
2016-12-14 14:54:40 +0000 Scope(Class[Openstack_tasks::Openstack_network::Routers]) (notice): MODULAR: openstack_network/routers.pp

Then I applied manually openstack_network/routers.pp and external_gateway was correctly configured:

Notice: /Stage[main]/Openstack_tasks::Openstack_network::Routers/Neutron_router[router04]/gateway_network_name: gateway_network_name changed '' to 'admin_floating_net'

neutron router-list
+--------------------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+
| id | name | external_gateway_info | distributed | ha |
+--------------------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+
| 8f46bb7e-7192-4aae-981e-daed4cbdd239 | router04 | {"network_id": "90252123-b344-42e2-a08c-06a039e1596b", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "e125106b-ebdc- | False | False |
| | | 4d4e-bb67-556c83da34f6", "ip_address": "10.109.39.131"}]} | | |
+--------------------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+

[0] https://github.com/openstack/fuel-library/blob/master/deployment/puppet/openstack_tasks/examples/openstack-network/tasks.yaml#L189-L228

[Nikita Karpin (mkarpin)]

Nope looks like I am wrong openstack_network/routers.pp should be applied only one time, I misread yaql expressions

[Nikita Karpin (mkarpin)]

also found that on envs where this is reproduced, during router creation only 2 from 3 neutron servers are up and there is no PUT request (for router external gateway setting in logs)

[Nikita Karpin (mkarpin)]

I found that in cases when this is reproduced, openstack_network/routers.pp is running on primary controller in the same time when openstack_network/plugins/ml2.pp and/or openstack_network/server_config.pp is running on other controllers. the thing is that openstack_network/plugins/ml2.pp and openstack_network/server_config.pp both are causing neutron-server restart.

So my assumption would be to place openstack_network/routers.pp after ml2 and server_config task over all controllers. I am testing it here - https://review.openstack.org/#/c/414011/

[Ivan Udovichenko (iudovichenko)]

The issue reproduced again:
http://paste.openstack.org/show/593185/

[Ivan Udovichenko (iudovichenko)]

Log in previous comment was noticed during https://ci.fuel-infra.org/job/11.0-community.main.ubuntu.bvt_2/210/console run.

Issues was also reproduced in during another https://ci.fuel-infra.org/job/11.0-community.main.ubuntu.bvt_2/218/console run:
http://paste.openstack.org/show/593190/

[Ivan Udovichenko (iudovichenko)]

And in another job https://ci.fuel-infra.org/job/11.0-community.main.ubuntu.bvt_2/229/console :
http://paste.openstack.org/show/593191/

[Ivan Udovichenko (iudovichenko)]

Same for https://product-ci.infra.mirantis.net/job/10.0.main.ubuntu.bvt_2/1056/console :
http://paste.openstack.org/show/593254/

[Dmitry Belyaninov (dbelyaninov)]

simillar fails
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron_destructive/162

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron_destructive_vxlan/160

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/415211

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/415212

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/414011
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=bd4438ff3f4530812447f69a97fc43f0c44df289
Submitter: Jenkins
Branch: master

commit bd4438ff3f4530812447f69a97fc43f0c44df289
Author: Mykyta Karpin <email address hidden>
Date: Thu Dec 22 12:29:54 2016 +0200

    Fix openstack networking task dependency

    Creation of networks and routers should always
    happen after neutron server has been fully configured
    on all openstack controllers.

    Closes-Bug: #1649627
    Change-Id: Idc194b65ba86b6673dcb339acf09e7cca44b6cc9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/415212
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=7bfe37b845267c7afee8ab8cfe38a3f231903a4f
Submitter: Jenkins
Branch: stable/mitaka

commit 7bfe37b845267c7afee8ab8cfe38a3f231903a4f
Author: Mykyta Karpin <email address hidden>
Date: Thu Dec 22 12:29:54 2016 +0200

    Fix openstack networking task dependency

    Creation of networks and routers should always
    happen after neutron server has been fully configured
    on all openstack controllers.

    Closes-Bug: #1649627
    Change-Id: Idc194b65ba86b6673dcb339acf09e7cca44b6cc9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/415211
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4a15eb71b987966421772d7edbe5c027870af435
Submitter: Jenkins
Branch: stable/newton

commit 4a15eb71b987966421772d7edbe5c027870af435
Author: Mykyta Karpin <email address hidden>
Date: Thu Dec 22 12:29:54 2016 +0200

    Fix openstack networking task dependency

    Creation of networks and routers should always
    happen after neutron server has been fully configured
    on all openstack controllers.

    Closes-Bug: #1649627
    Change-Id: Idc194b65ba86b6673dcb339acf09e7cca44b6cc9

[Ilya Bumarskov (ibumarskov)]

TC "neutron_vlan_ha" for Fuel 9.2 snapshot-id #801 (RC1) is passed - https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron/184/

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.