Public VIP is inaccessible from external networks

[Vasyl Saienko]

Failed to access to horizon at VIP ip from external network.

[root@nailgun ~]# fuel --f
DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient.
api: '1.0'
astute_sha: 3e43895621370e65737b240e999f6c7391e68352
auth_required: true
build_id: 2015-03-19_07-51-55
build_number: '162'
feature_groups:
- experimental
fuellib_sha: de7ea04ef1a8b8399949d1ad28c6ac29d2b3e084
fuelmain_sha: fbfe367850a7e902d67d5ba05bfb9ed937a38dae
nailgun_sha: ff2e37abf5801757bd20b80a7d2d41df2e08ed63
ostf_sha: 9435ec9f0aaf5bb8b9773347af871ab281b9ae78
production: docker
python-fuelclient_sha: 14f2b3e0c9fd1580d85a83b7eea95fbe893665ac
release: '6.1'
release_versions:
  2014.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 3e43895621370e65737b240e999f6c7391e68352
      build_id: 2015-03-19_07-51-55
      build_number: '162'
      feature_groups:
      - experimental
      fuellib_sha: de7ea04ef1a8b8399949d1ad28c6ac29d2b3e084
      fuelmain_sha: fbfe367850a7e902d67d5ba05bfb9ed937a38dae
      nailgun_sha: ff2e37abf5801757bd20b80a7d2d41df2e08ed63
      ostf_sha: 9435ec9f0aaf5bb8b9773347af871ab281b9ae78
      production: docker
      python-fuelclient_sha: 14f2b3e0c9fd1580d85a83b7eea95fbe893665ac
      release: '6.1'

Ubuntu, 5 nodes (3 controller + 2 Computes) ALL parameters by default.

[Vasyl Saienko (vsaienko)]

How to reproduce. I have a the following setup

my_pc (172.18.214.24) <------ MIRANTIS NETWORK -------> fuel172.16.56.98, horizon VIP 172.16.56.99, controller1 172.16.56.101, controller2 172.16.56.102 controller3 172.16.56.103

I can access 172.16.56.101:80 from my pc

$ telnet 172.16.56.101 80
Trying 172.16.56.101...
Connected to 172.16.56.101.
Escape character is '^]'.
^C^]
telnet> ^Cq
Connection closed.

but can't horizon VIP:

vsaienko@vsaienko-pc:~/Work/review.fuel-infra.org/openstack/oslo.messaging$ telnet 172.16.56.99 80
Trying 172.16.56.99...

at the same time, packets can reach to controller, but no responce from it

root@node-1:~# tcpdump -nei br-ex host 172.18.214.24
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-ex, link-type EN10MB (Ethernet), capture size 65535 bytes
14:17:21.197000 52:54:00:1d:70:70 > d6:0e:51:5f:24:da, ethertype IPv4 (0x0800), length 74: 172.18.214.24.53300 > 172.16.56.99.80: Flags [S], seq 3542400440, win 29200, options [mss 1460,sackOK,TS val 26648494 ecr 0,nop,wscale 7], length 0
14:17:22.195110 52:54:00:1d:70:70 > d6:0e:51:5f:24:da, ethertype IPv4 (0x0800), length 74: 172.18.214.24.53300 > 172.16.56.99.80: Flags [S], seq 3542400440, win 29200, options [mss 1460,sackOK,TS val 26648744 ecr 0,nop,wscale 7], length 0
14:17:24.200121 52:54:00:1d:70:70 > d6:0e:51:5f:24:da, ethertype IPv4 (0x0800), length 74: 172.18.214.24.53300 > 172.16.56.99.80: Flags [S], seq 3542400440, win 29200, options [mss 1460,sackOK,TS val 26649245 ecr 0,nop,wscale 7], length 0

As possible solution it is switch default route in haproxy namespace

root@node-1:~# ip netns exec haproxy ip route del default
root@node-1:~# ip netns exec haproxy ip route add default via 172.16.56.97

root@node-1:~# tcpdump -nei br-ex host 172.18.214.24
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-ex, link-type EN10MB (Ethernet), capture size 65535 bytes
14:21:33.274221 52:54:00:1d:70:70 > d6:0e:51:5f:24:da, ethertype IPv4 (0x0800), length 74: 172.18.214.24.53378 > 172.16.56.99.80: Flags [S], seq 654135808, win 29200, options [mss 1460,sackOK,TS val 26711514 ecr 0,nop,wscale 7], length 0
14:21:33.274313 d6:0e:51:5f:24:da > 52:54:00:1d:70:70, ethertype IPv4 (0x0800), length 74: 172.16.56.99.80 > 172.18.214.24.53378: Flags [S.], seq 2771456757, ack 654135809, win 28960, options [mss 1460,sackOK,TS val 1673755 ecr 26711514,nop,wscale 7], length 0
14:21:33.361838 52:54:00:1d:70:70 > d6:0e:51:5f:24:da, ethertype IPv4 (0x0800), length 66: 172.18.214.24.53378 > 172.16.56.99.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 26711535 ecr 1673755], length 0
14:21:33.816452 52:54:00:1d:70:70 > d6:0e:51:5f:24:da, ethertype IPv4 (0x0800), length 71: 172.18.214.24.53378 > 172.16.56.99.80: Flags [P.], seq 1:6, ack 1, win 229, options [nop,nop,TS val 26711649 ecr 1673755], length 5
14:21:33.816597 d6:0e:51:5f:24:da > 52:54:00:1d:70:70, ethertype IPv4 (0x0800), length 253: 172.16.56.99.80 > 172.18.214.24.53378: Flags [F.], seq 1:188, ack 6, win 227, options [nop,nop,TS val 1673890 ecr 26711649], length 187
14:21:33.892212 52:54:00:1d:70:70 > d6:0e:51:5f:24:da, ethertype IPv4 (0x0...

[Bogdan Dobrelya (bogdando)]

The issue is that gateway specified for the deployment ( 172.16.56.97 ) was not properly set inside haproxy netns of the controller node hosting the public VIP:

root@node-1:~# ip netns exec haproxy ip ro
default via 240.0.0.1 dev hapr-p

but it should be:
root@node-1:~# ip netns exec haproxy ip ro
default via 172.16.56.97 dev hapr-p

[Sergey Kolekonov (skolekonov)]

I've also found out that Attribute "gateway" for Pacemaker resource vip__public is empty, but it should contain the appropriare gateway.
I've just executed:
pcs resource update vip__public gateway=<my gateway>
and vip is available from the outside (including Horizon)

[Bogdan Dobrelya (bogdando)]

Do you have it empty for Neutron deployments?
As for Nova-network, I have it set to "link", for example:
primitive vip__public ocf:fuel:ns_IPaddr2 \
...
gateway_metric=10 ip=10.109.1.4 ns=haproxy gateway=link
...

[Sergey Kolekonov (skolekonov)]

Yes, it was Neutron deployment