Internet doesn't get forwarded via master node

[Mike Scherbakov]

ISO #302
fuellib_sha: "e9c3ba332b05120c967b20260c7b223afc1b4f1a"
After installation of Fuel Master using vbox scripts, I'm not able to ping Internet from boostrap nodes. I see that DNS resolution works just fine, and I can see traffic coming to the Master node and even goes out from the right Interface (eth2, which is configured in Vbox as NAT).

The problem is that traffic doesn't go back to the bootstrap node.

I've found following rules in POSTROUTING on master node:
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE tcp -- 10.20.0.0/24 0.0.0.0/0 /* 004 forward_admin_net */
MASQUERADE all -- 172.17.0.0/16 0.0.0.0/0

When I changed the first rule to:
MASQUERADE all -- 10.20.0.0/24 !10.20.0.0/24
via iptables -t nat -A POSTROUTING -s 10.20.0.0/24 \! -d 10.20.0.0/24 -j MASQUERADE, and deleting the original rule, traffic started to pass.

Please research if it's really the original issue. iptables rule is being created in deployment/puppet/nailgun/manifests/iptables.pp.

[Mike Scherbakov (mihgen)]

I've increased priority to High, as this affects Ubuntu provisioning for me on VBox demo installation.

[Nastya Urlapova (aurlapova)]

Guys, if this issue goes with High priority we have to fix it in 6.1.

[Oleksiy Molchanov (omolchanov)]

We have checked this issue with Anastasiya Palkina and didn't faced such problems.

Earlier we had an issue that you must set sysctl net.bridge.bridge-nf-call-iptables=0 on your hypervisor, but I am not sure that it suits this case.

Anyway I am lowering the priority, because it is related to your env.

[Oleksiy Molchanov (omolchanov)]

Just checked, sysctl rule won't help you.

[Oleksiy Molchanov (omolchanov)]

Disregard my previous messages, my bad. Seems that it is a duplicate to 1442988

[Mike Scherbakov (mihgen)]

This is *NOT* duplicate of 1442988. This bug is that you can't reach Internet, if you run ping from bootstrap nodes. 1442988 is that you can't reach Internet, if you run ping after OS is installed and l23network is ran (and reconfigured default gateway).

I'm running on Mac with VBox scripts, and this is repeatedly reproduced. Please do not lower the priority of this issue.

If it does work on Nastya's env, please share what version of virtualbox scripts she is using (I'm running on fuel-main repo master). I'll ask some other folks running on Mac to try to reproduce same issue.

Also, I've provided my doubts on correctness of iptables rules on master node. Please comment. As I stated in description, with changed iptables rule I was able to fix the issue.

[Serhii Ovsianikov (sovsianikov)]

Mike, guys

By default we have internet connectivity on the slave nodes in bootstrap (I mean vbox). We can check it using the following command " curl ya.ru " (for example).

For the ping command I added iptables rule in the next review https://review.openstack.org/#/c/173768

I checked it and I have ping from the slave nodes in bootstrap after the fuel install.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/176250

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/176250
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=53802ae041187374560e8bbcc0badb29be3d4fa4
Submitter: Jenkins
Branch: master

commit 53802ae041187374560e8bbcc0badb29be3d4fa4
Author: Oleksiy Molchanov <email address hidden>
Date: Wed Apr 22 13:38:24 2015 +0300

    Add possibility to nat all proto

    Add possibility to nat all kind of protocols
    on master node

    Change-Id: I1079e9c2da2dee64bb9949fc08de4eaf4e194677
    Closes-Bug: 1445124

[Anastasia Palkina (apalkina)]

Verified on ISO #414 installed on Ubuntu

"build_id": "2015-05-13_22-55-26", "build_number": "414", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-05-13_22-55-26", "build_number": "414", "api": "1.0", "fuel-library_sha": "f9310aa52628d70b45a8d9cb735dd309f9b556f8", "nailgun_sha": "afbcea1d4928e588a8858e154770cd9d031a1fc7", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "e4ebbc720c2d5e4546b51758c5796821ed4377f6", "astute_sha": "484d5038b81b7260597be027b1265dc88091026b", "fuel-ostf_sha": "1ae384a28b42d5325bd5d2546aead1e9d502f6a8", "release": "6.1", "fuelmain_sha": "1c03fb884d3e79165a2406d00a5ea8679e35f4f7"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "f9310aa52628d70b45a8d9cb735dd309f9b556f8", "nailgun_sha": "afbcea1d4928e588a8858e154770cd9d031a1fc7", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "e4ebbc720c2d5e4546b51758c5796821ed4377f6", "astute_sha": "484d5038b81b7260597be027b1265dc88091026b", "fuel-ostf_sha": "1ae384a28b42d5325bd5d2546aead1e9d502f6a8", "release": "6.1", "fuelmain_sha": "1c03fb884d3e79165a2406d00a5ea8679e35f4f7"

[Mike Scherbakov (mihgen)]

I've reopened this bug for 8.0.
I'm trying RC3 7.0 against stable/7.0 vbox scripts, and experience the same issue. icmp doesn't pass, while telnet ya.ru 80 is just Ok.

[Matthew Mosesohn (raytrac3r)]

This isn't reproduced on Ubuntu or Fedora hosts. We need access to test on a Mac where this is reproducing. Back to fuel-library for now.

[Stanislaw Bogatkin (sbogatkin)]

Seems that problem appears only for Mac users, cause all works okay on linux box.

[Stanislaw Bogatkin (sbogatkin)]

So, need test from someone with OSX OS.

[Maksim Malchuk (mmalchuk)]

I'm trying RC3 7.0 against stable/7.0 vbox scripts, on MacOS X Yosemite 10.10.5 and issue didn't reproduced. icmp and tcp to ya.ru works fine.

[Maksim Malchuk (mmalchuk)]

proof screenshot from my Mac

[Maksim Malchuk (mmalchuk)]

some more proof from tcpdump. i've run ping ya.ru from 10.20.0.6 node.

[root@fuel ~]# tcpdump -qn -i eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:05:14.402628 IP 10.20.0.6 > 213.180.204.3: ICMP echo request, id 15899, seq 43, length 64
14:05:14.425795 IP 213.180.204.3 > 10.20.0.6: ICMP echo reply, id 15899, seq 43, length 64

[root@fuel ~]# tcpdump -qn -i eth2 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes
14:05:20.412683 IP 192.168.200.15 > 213.180.204.3: ICMP echo request, id 15899, seq 49, length 64
14:05:20.435058 IP 213.180.204.3 > 192.168.200.15: ICMP echo reply, id 15899, seq 49, length 64

so NAT (MASQUERADE) works fine