Public VIP is inaccessible from external networks

[Victor Ryzhenkin]

ISO: {"build_id": "2015-03-11_21-47-59", "ostf_sha": "ecb8e294b0acbdc5b0300d5e39028fb26ecc9088", "build_number": "187", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-11_21-47-59", "ostf_sha": "ecb8e294b0acbdc5b0300d5e39028fb26ecc9088", "build_number": "187", "api": "1.0", "nailgun_sha": "a720a2da99690eb2d2c19ddc5d739384312a8ac2", "production": "docker", "python-fuelclient_sha": "0f4ca9c2798da34797dd082130d22cac04c998a9", "astute_sha": "5cdd4ae4037aa29f4c876d441af15cad82f5a6cb", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "0791400dd8224647ff9a5cb8051ce82b2c8863b1", "fuellib_sha": "cfdfcbdb0197f606b4c93e6dd4011525df9a3ff8"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "a720a2da99690eb2d2c19ddc5d739384312a8ac2", "production": "docker", "python-fuelclient_sha": "0f4ca9c2798da34797dd082130d22cac04c998a9", "astute_sha": "5cdd4ae4037aa29f4c876d441af15cad82f5a6cb", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "0791400dd8224647ff9a5cb8051ce82b2c8863b1", "fuellib_sha": "cfdfcbdb0197f606b4c93e6dd4011525df9a3ff8"}

STEPS:
1. Deploy OS-neutron GRE, HA (3 Virtual Controllers, 1 Virtual Compute)
2. Log into Horizon.

Expected:
Horizon UI accesible via external network

Actual result:
Horizon UI accesible only from controller nodes.

NOTES:
I can't find any logs with failures of network, but in one case when i executed "pcs resource",
resource vip__public was in FAILURE state.
In Iptables i can't find any information about ip of haproxy, but resource vip__public should provide this information to iptables.

I tried to upload snapshot, but to my amazement it was over 200M. I'll take it on my Dropbox later.

[Victor Ryzhenkin (vryzhenkin)]

Snapshot:
https://www.dropbox.com/s/6d2w6z8vc3k8q8l/fuel-snapshot-2015-03-12_19-39-31.tgz?dl=0

[Stanislav Makar (smakar)]

Could you please tell us how you deployed this env (e.g. manually, using dos.py, or virtualbox scripts )?

[Stanislav Makar (smakar)]

related bug: https://bugs.launchpad.net/fuel/+bug/1396126

looks like it is due to above bug

[Victor Ryzhenkin (vryzhenkin)]

Stanislav, this env was deployed manually.

More information:
We create ssh tunnel to access Horizon.
Our public router was alive.
We deployed instance and we has access to this instance from external network and it was succesfully pinged.
Instance was in the same network with controllers.

Also, for debugging, we shutdown our controllers, and create test VMs. We have configured networking on them, and boot to bootstrap.
After that, we configure public network on VMs and add ip from every controller to this VMs. Every VM was successfully pinged from our external network.

[Victor Ryzhenkin (vryzhenkin)]

Stanistav, in our case deployment was finished successfully.. Not failed.

[Bogdan Dobrelya (bogdando)]

I lowered the bug priority as it doesn't affect the deployment, please update the status back if you think it should be higher

[Victor Ryzhenkin (vryzhenkin)]

Bogdan, i'm think that this bug should be with high priority at least.

[Timur Nurlygayanov (tnurlygayanov)]

Bogdan, if Horizon dashboard is not available after the deployment it is Critical bug because we can not release the product with this issue.

Importance changed to High, we need to track this issue and fix it as soon as possible, it is blocker for QA team and for release.

[Bogdan Dobrelya (bogdando)]

QA team, please confirm either this issue affects other releases

[Yaroslav Lobankov (ylobankov)]

I have never encountered this issue in 6.0.1. I have deployed the latest ISO (#133) . The issue has not been reproduced.

[Bogdan Dobrelya (bogdando)]

The RC of this issue is found here https://bugs.launchpad.net/mos/+bug/1434088

[Bogdan Dobrelya (bogdando)]

This issue should be related to the 6.1 specific changes which came with the networking refactoring

[Bogdan Dobrelya (bogdando)]

Sergey Kolekonov (skolekonov) wrote 13 minutes ago:
I've also found out that Attribute "gateway" for Pacemaker resource vip__public is empty, but it should contain the appropriare gateway.
I've just executed:
pcs resource update vip__public gateway=<my gateway>
and vip is available from the outside (including Horizon)

[Bogdan Dobrelya (bogdando)]

I'm not sure either replacing 240.0.0.1 route to gateway IP is an appropriate solution, but it works. Also I noticed that affected environments do not have a scope link route inside of haproxy namespace, which is:

default dev hapr-p scope link metric 10

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/166273

[Oleksiy Molchanov (omolchanov)]

Please try the latest ISO.

[Victor Ryzhenkin (vryzhenkin)]

Still reproduced with latest iso.

[Vadim Rovachev (vrovachev)]

Patch and bug verified on:

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  api: "1.0"
  build_number: "233"
  build_id: "2015-03-26_21-32-43"
  nailgun_sha: "b163f6fc77d6639aaffd9dd992e1ad96951c3bbf"
  python-fuelclient_sha: "e5e8389d8d481561a4d7107a99daae07c6ec5177"
  astute_sha: "3f1ece0318e5e93eaf48802fefabf512ca1dce40"
  fuellib_sha: "9c7716bc2ce6075065d7d9dcf96f4c94662c0b56"
  ostf_sha: "a4cf5f218c6aea98105b10c97a4aed8115c15867"
  fuelmain_sha: "320b5f46fc1b2798f9e86ed7df51d3bda1686c10"

Without patch bug reproduced.
With patch bug not reproduced.

Patch works.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/166273
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=8e76bdeb583fac6ee7b3769b2754831dcff871e6
Submitter: Jenkins
Branch: master

commit 8e76bdeb583fac6ee7b3769b2754831dcff871e6
Author: Vladimir Kuklin <email address hidden>
Date: Fri Apr 3 02:50:16 2015 +0300

    Revert of gateways for VIPs

    This commit sets default route for public interfaces
    and also sets static routes for haproxy namespaces to
    the networks which are connected to the host directly
    with metric of 10000 to ensure that these routes do
    not override routes installed by kernel on interfaces
    configuration stage

    Regression caused by fix of
    https://bugs.launchpad.net/fuel/+bug/1433150

    Change-Id: I7df8efad4aa119896314e77c29d9b024599dc865
    Closes-Bug: 1431547

[Vadim Rovachev (vrovachev)]

verified on:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  api: "1.0"
  build_number: "281"
  build_id: "2015-04-06_03-15-13"
  nailgun_sha: "a2dc04af9c5838a990adeab81789d4c090943701"
  python-fuelclient_sha: "9ad42671130deb2cc81135fb453ef0793883be2c"
  astute_sha: "240bfefe0a0b5af21b79854d0c96d8a3ee0fd885"
  fuellib_sha: "601cc8e1dccbdcf328f83f071270609e824d6c5a"
  ostf_sha: "b978badb43243d3f1b85dde15b8e301a21f12254"
  fuelmain_sha: "f90d7d40ceff39ee6d7587b183c63bd00ed0743f"