[vbox] No Internet connection on compute nodes in HA mode when run under VBox

[Timur Nurlygayanov]

Steps To Reproduce:
1. Deploy Fuel 4.1 using scripts for Virtual Box
2. Deploy OpenStack with HA mode, Cent OS, 3 controller and 1 compute, Neutron with VLANs (and install Murano, Savanna and Ceilometer).
3. Login to compute node and ping mail.ru: ping mail.ru
4. Create VM in OpenStack and ping 8.8.8.8 or mail.ru from this host

Observed Result:
We can not ping any hosts in Internet from compute nodes and VMs.
We can see also that name resolving works good (wen we ping mail.ru we know th IP address of this host)

[root@node-2 ~]# ping mail.ru
PING mail.ru (217.69.139.199) 56(84) bytes of data.
^C
--- mail.ru ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2936ms

[root@node-2 ~]# cat /etc/resolv.conf
# Generated by NetworkManager
domain domain.tld
search domain.tld
nameserver 10.20.0.2

[root@node-2 ~]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.20.0.0 * 255.255.255.0 U 0 0 0 br-fw-admin
192.168.1.0 * 255.255.255.0 U 0 0 0 br-storage
172.16.0.0 * 255.255.255.0 U 0 0 0 br-ex
192.168.0.0 * 255.255.255.0 U 0 0 0 br-mgmt
192.168.122.0 * 255.255.255.0 U 0 0 0 virbr0
link-local * 255.255.0.0 U 1002 0 0 eth0
link-local * 255.255.0.0 U 1003 0 0 eth1
link-local * 255.255.0.0 U 1004 0 0 eth2
link-local * 255.255.0.0 U 1009 0 0 br-ex
link-local * 255.255.0.0 U 1010 0 0 br-mgmt
link-local * 255.255.0.0 U 1011 0 0 br-storage
link-local * 255.255.0.0 U 1012 0 0 br-fw-admin
link-local * 255.255.0.0 U 1013 0 0 br-prv
default 172.16.0.1 0.0.0.0 UG 0 0 0 br-ex

[root@node-2 ~]# tracepath mail.ru
 1: 172.16.0.6 (172.16.0.6) 0.185ms pmtu 1500
 1: 172.16.0.1 (172.16.0.1) 1.498ms
 1: 172.16.0.1 (172.16.0.1) 0.305ms
 2: no reply
 3: no reply
 4: no reply
 5: no reply

[Timur Nurlygayanov (tnurlygayanov)]

10.20.0.2 in this case it is private IP of Fuel Master node.

[Timur Nurlygayanov (tnurlygayanov)]

Steps How We can fix this problem:

in Virtual Box need to change the type of network to NAT and execute on the host machine (where Virtual Box is running) the following command:

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

where eth0 - the interface with the access to the Internet.

[Mike Scherbakov (mihgen)]

Lowering priority: issue exists only if you run setup under VirtualBox scripts due to configuration of networking on the host machine.
Suggested
> iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
obviously won't work for Mac users.

[Miroslav Anashkin (manashkin)]

Working network settings for MacOS overview available here:
https://drive.google.com/file/d/0BwNB1Fj6mQMBRnJHYjBtSXlIajQ/edit?usp=sharing

To set up NAT for VBox host-only adapter under MacOS one may do the following (all under root account!)
1.
`sysctl -w net.inet.ip.forwarding=1`

2. Do not use "ipfw" — use "pf" only.
Below is the pf config example with the necessary rules declared. Simply add the similar settings to your pf.conf:

`# cat /etc/pf.conf`
#######################
scrub-anchor "com.apple/*"
nat-anchor "com.apple/*"
rdr-anchor "com.apple/*"
nat on en0 inet from ! (en0) to any -> (en0)
dummynet-anchor "com.apple/*"
anchor "com.apple/*"
load anchor "com.apple" from "/etc/pf.anchors/com.apple"
pass in on vboxnet0
pass in on vboxnet1
pass in on vboxnet2
pass in on vboxnet3
pass in on vboxnet4
pass in on vboxnet5
pass in on vboxnet6
pass in on vboxnet7
pass in on vboxnet8
pass in on vboxnet9
pass in on vboxnet10
pass in on vboxnet11
pass in on vboxnet12
pass in on vboxnet13
pass in on vboxnet14
pass in on vboxnet15
#######################

3. Apply the settings:
`pfctl -f /etc/pf.conf`

The key rules in the settings listed above are `nat on en0 inet ...`
and all that "pass" rules, all the remained rules are mostly defaults.

[Fabrizio Soppelsa (fsoppelsa)]

Some minor issues on setting up /etc/pf.conf: our rules must be in precise order, otherwise they won't be accepted by `pfctl`.

In /etc/pf.conf there's already a minimal configuration, on brand-new installations:

scrub-anchor "com.apple/*"
nat-anchor "com.apple/*"
rdr-anchor "com.apple/*"
dummynet-anchor "com.apple/*"
anchor "com.apple/*"
load anchor "com.apple" from "/etc/pf.anchors/com.apple"

For instance, one should inject

nat on en0 inet from ! (en0) to any -> (en0)

Between ^dummynet and ^anchor

Keeping in mind that this file won't be likely modified by the desktop user and probably he won't try Fuel on production (with production defined /etc/pf.conf), would you suggest to simply backup /etc/pf.conf to /etc/pf.conf.bak, reload PF rules, run the Fuel demo and finally restore /etc/pf.conf.bak in place?

Other suggestions work like a charm.

Also, solving this bug means that ./launch*.sh must be run by root.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/119717

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/120429

[OpenStack Infra (hudson-openstack)]

Change abandoned by Fabrizio Soppelsa (<email address hidden>) on branch: master
Review: https://review.openstack.org/120429
Reason: Duplicate

[OpenStack Infra (hudson-openstack)]

Change abandoned by Fabrizio Soppelsa (<email address hidden>) on branch: master
Review: https://review.openstack.org/119717
Reason: Duplicate

[OpenStack Infra (hudson-openstack)]

Change abandoned by Fabrizio Soppelsa (<email address hidden>) on branch: master
Review: https://review.openstack.org/119717
Reason: Merge conflict

[Serhii Ovsianikov (sovsianikov)]

There is similar bug https://bugs.launchpad.net/fuel/+bug/1442988 and review https://review.openstack.org/#/c/173768

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/119717
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=deda3b648409aeba6ceada212f2c27764fc1915d
Submitter: Jenkins
Branch: master

commit deda3b648409aeba6ceada212f2c27764fc1915d
Author: Serhiy Ovsianikov <email address hidden>
Date: Thu May 14 20:20:18 2015 +0300

    Fix for VBox "No Internet connection on compute nodes"

    The scripts work without any firewall tuning on the user's computers.
    The Fuel master node is forwarding the public network through
    the existing NAT interface.

    Change-Id: I2962bccb39828da4ff011889f990a67d89322ca3
    Closes-Bug: #1288135