OpenStack Compute (nova)

iptables drops forward rules for hosts

Bug #890195 reported by Jay Lee
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Vish Ishaya
OpenStack Compute (nova) 2012.1 "essex"
Diablo
Fix Released
Undecided
Unassigned
OpenStack Compute (nova) 2011.3.1

Bug Description

Problem)
This is my multi_host configuration.
Host: nova-1, nova-2
instnace: instance-1(10.0.149.4, nova-1), instance-2(10.0.149.6nova-2)
bridge: br100(10.0.149.3, nova-1), br100(10.0.149.5, nova-2)

when I try to ping instance-1 at nova-2. any reply didn't come.
and also ping instance-2 at nova-1, result is same.

but, ping to bridge(10.0.149.3,nova-1) from nova-2 is ok.
ping to bridge(10.0.149.5,nova-2) from nova-1 is also ok.

so, I check iptables.
This is rule of cause of problem.
# iptables -L
....
Chain nova-network-FORWARD (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
....
so, I drop this rule,
# iptables -D nova-network-FORWARD 2
# iptables -D nova-network-FORWARD 1

then,
any instance can ping and ssh to other instance.

this rule add when nova-network start or new instance become boot.

this is code of above rule.

nova.network.linux_net.py
 997 if gateway:
 998 iptables_manager.ipv4['filter'].add_rule('FORWARD',
 999 '--in-interface %s -j ACCEPT' % \
1000 bridge)
1001 iptables_manager.ipv4['filter'].add_rule('FORWARD',
1002 '--out-interface %s -j ACCEPT' % \
1003 bridge)
1004 # else:
1005 # iptables_manager.ipv4['filter'].add_rule('FORWARD',
1006 # '--in-interface %s -j DROP' % \
1007 # bridge)
1008 # iptables_manager.ipv4['filter'].add_rule('FORWARD',
1009 # '--out-interface %s -j DROP' % \
1010 # bridge)
1011 #

I modified 1004 ~ 1010 lines. I commented out that else: routine and I restart nova-network module.

then, above rule didn't appears.

# iptables -L
...
Chain nova-network-FORWARD (1 references)
target prot opt source destination
...

I think instance can connect any host in multi_host configuration. so that rule should be modified.

Jay Lee (hyangii)
tags: added: iptables
Revision history for this message
Vish Ishaya (vishvananda) wrote : Re: [Bug 890195] [NEW] iptables problem: can't connect from other host to instance
Download full text (5.5 KiB)

VMs aren't necessarily supposed to be pingable from hosts. They should be pingable from the network host (or any host if using a --multi_host network) if the flag --allow_same_net_traffic is true. Otherwise you need to specifically allow connections via security group rules.

On Nov 14, 2011, at 4:39 AM, jaesanglee wrote:

> Public bug reported:
>
> Problem)
> This is my multi_host configuration.
> Host: nova-1, nova-2
> instnace: instance-1(10.0.149.4, nova-1), instance-2(10.0.149.6nova-2)
> bridge: br100(10.0.149.3, nova-1), br100(10.0.149.5, nova-2)
>
> when I try to ping instance-1 at nova-2. any reply didn't come.
> and also ping instance-2 at nova-1, result is same.
>
> but, ping to bridge(10.0.149.3,nova-1) from nova-2 is ok.
> ping to bridge(10.0.149.5,nova-2) from nova-1 is also ok.
>
> so, I check iptables.
> This is rule of cause of problem.
> # iptables -L
> ....
> Chain nova-network-FORWARD (1 references)
> target prot opt source destination
> DROP all -- anywhere anywhere
> DROP all -- anywhere anywhere
> ....
> so, I drop this rule,
> # iptables -D nova-network-FORWARD 2
> # iptables -D nova-network-FORWARD 1
>
> then,
> any instance can ping and ssh to other instance.
>
> this rule add when nova-network start or new instance become boot.
>
> this is code of above rule.
>
> nova.network.linux_net.py
> 997 if gateway:
> 998 iptables_manager.ipv4['filter'].add_rule('FORWARD',
> 999 '--in-interface %s -j ACCEPT' % \
> 1000 bridge)
> 1001 iptables_manager.ipv4['filter'].add_rule('FORWARD',
> 1002 '--out-interface %s -j ACCEPT' % \
> 1003 bridge)
> 1004 # else:
> 1005 # iptables_manager.ipv4['filter'].add_rule('FORWARD',
> 1006 # '--in-interface %s -j DROP' % \
> 1007 # bridge)
> 1008 # iptables_manager.ipv4['filter'].add_rule('FORWARD',
> 1009 # '--out-interface %s -j DROP' % \
> 1010 # bridge)
> 1011 #
>
> I modified 1004 ~ 1010 lines. I commented out that else: routine and I
> restart nova-network module.
>
> then, above rule didn't appears.
>
> # iptables -L
> ...
> Chain nova-network-FORWARD (1 references)
> target prot opt source destination
> ...
>
> I think instance can connect any host in multi_host configuration. so
> that rule should be modified.
>
> ** Affects: nova
> Importance: Undecided
> Status: New
>
> --
> You received this bug notification because you are subscribed to
> OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/890195
>
> Title:
> iptables problem: can't connect from other host to instance
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> Problem)
> This is my multi_host configuration.
> Host: nova-1, nova-2
> instnace: instance-1(10.0.149.4, nova-1), i...

Read more...

Revision history for this message
Jay Lee (hyangii) wrote : Re: iptables problem: can't connect from other host to instance
Download full text (3.5 KiB)

Ok, I have one more problem with above code.
Now,
when I start nova-network, iptables added this rule.
Chain nova-network-FORWARD (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere

because In nova/network/linux_net.py,
 995 # Don't forward traffic unless we were told to be a gateway
 996 if gateway:
 997 iptables_manager.ipv4['filter'].add_rule('FORWARD',
 998 '--in-interface %s -j ACCEPT' % \
 999 bridge)
1000 iptables_manager.ipv4['filter'].add_rule('FORWARD',
1001 '--out-interface %s -j ACCEPT' % \
1002 bridge)
1003
1004 else:
1005 iptables_manager.ipv4['filter'].add_rule('FORWARD',
1006 '--in-interface %s -j DROP' % \
1007 bridge)
1008 iptables_manager.ipv4['filter'].add_rule('FORWARD',
1009 '--out-interface %s -j DROP' % \
1010 bridge)

above this code, gateway set 'None' and 'gateway = fields[1]'
 967 gateway = None
 968 out, err = _execute('route', '-n', run_as_root=True)
 969 for line in out.split('\n'):
 970 fields = line.split()
 971 if fields and fields[0] == '0.0.0.0' and \
 972 fields[-1] == interface:
 973 gateway = fields[1]
 974 _execute('route', 'del', 'default', 'gw', gateway,
 975 'dev', interface, check_exit_code=False,
 976 run_as_root=True)

# route -n
...
0.0.0.0 192.168.0.1 0.0.0.0 UG 100 0 0 eth1

but, my nova.conf
--public_interface=eth1
--flat_interface=eth2

so,
 fields[-1] = eth1
 interface = eth2
fileds[-1] != interface, so gateway didn't set.

finally below rule run.
1004 else:
1005 iptables_manager.ipv4['filter'].add_rule('FORWARD',
1006 '--in-interface %s -j DROP' % \
1007 bridge)
1008 iptables_manager.ipv4['filter'].add_rule('FORWARD',
1009 '--out-interface %s -j DROP' % \
1010 bridge)

As a result, below rule added.
Chain nova-network-FORWARD (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere

and, In a instance, I can't connect to any other site and host.

# nova list
+--------------------------------------+---------+--------+---------------------+
| ID | Name | Status | Networks |
 +--------------------------------------+---------+--------+---------------------+
| bbfa18ce-a21a-460c-8689-e746e0583939 | backend | ACTIVE | private=10.0.152.21 |
 +--------------------------------------+...

Read more...

Revision history for this message
Vish Ishaya (vishvananda) wrote : Re: [Bug 890195] iptables problem: can't connect from other host to instance
Download full text (6.5 KiB)

That is definitely a bug.

The inner gateway is hiding the outer one.

Vish

On Nov 16, 2011, at 1:04 AM, jaesanglee wrote:

> Ok, I have one more problem with above code.
> Now,
> when I start nova-network, iptables added this rule.
> Chain nova-network-FORWARD (1 references)
> target prot opt source destination
> DROP all -- anywhere anywhere
> DROP all -- anywhere anywhere
>
> because In nova/network/linux_net.py,
> 995 # Don't forward traffic unless we were told to be a gateway
> 996 if gateway:
> 997 iptables_manager.ipv4['filter'].add_rule('FORWARD',
> 998 '--in-interface %s -j ACCEPT' % \
> 999 bridge)
> 1000 iptables_manager.ipv4['filter'].add_rule('FORWARD',
> 1001 '--out-interface %s -j ACCEPT' % \
> 1002 bridge)
> 1003
> 1004 else:
> 1005 iptables_manager.ipv4['filter'].add_rule('FORWARD',
> 1006 '--in-interface %s -j DROP' % \
> 1007 bridge)
> 1008 iptables_manager.ipv4['filter'].add_rule('FORWARD',
> 1009 '--out-interface %s -j DROP' % \
> 1010 bridge)
>
> above this code, gateway set 'None' and 'gateway = fields[1]'
> 967 gateway = None
> 968 out, err = _execute('route', '-n', run_as_root=True)
> 969 for line in out.split('\n'):
> 970 fields = line.split()
> 971 if fields and fields[0] == '0.0.0.0' and \
> 972 fields[-1] == interface:
> 973 gateway = fields[1]
> 974 _execute('route', 'del', 'default', 'gw', gateway,
> 975 'dev', interface, check_exit_code=False,
> 976 run_as_root=True)
>
> # route -n
> ...
> 0.0.0.0 192.168.0.1 0.0.0.0 UG 100 0 0 eth1
>
> but, my nova.conf
> --public_interface=eth1
> --flat_interface=eth2
>
> so,
> fields[-1] = eth1
> interface = eth2
> fileds[-1] != interface, so gateway didn't set.
>
> finally below rule run.
> 1004 else:
> 1005 iptables_manager.ipv4['filter'].add_rule('FORWARD',
> 1006 '--in-interface %s -j DROP' % \
> 1007 bridge)
> 1008 iptables_manager.ipv4['filter'].add_rule('FORWARD',
> 1009 '--out-interface %s -j DROP' % \
> 1010 bridge)
>
> As a result, below rule added.
> Chain nova-network-FORWARD (1 references)
> target prot opt source destination
> DROP all -- anywhere anywhere
> DROP all -- anywhere anywhere
>
> and, In a instance, I can't connect to any other site and host.
>
> # nova list
> +--------------------------------------+---------+--------+---------------------+
> | ID ...

Read more...

Changed in nova:
status: New → In Progress
importance: Undecided → High
assignee: nobody → Vish Ishaya (vishvananda)
Vish Ishaya (vishvananda)
summary: - iptables problem: can't connect from other host to instance
+ iptables drops forward rules for hosts
Changed in nova:
milestone: none → essex-2
Revision history for this message
Openstack Gerrit (openstack-gerrit) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/1694
Committed: http://github.com/openstack/nova/commit/41af372219793556e6ba335d765761fa277107df
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit 41af372219793556e6ba335d765761fa277107df
Author: Vishvananda Ishaya <email address hidden>
Date: Wed Nov 16 10:17:23 2011 -0800

    Makes sure gateways forward properly

     * Fixes bug 890195
     * Fixes missing context in dhcp call
     * Adds test to verify call is correct

    Change-Id: Ic099082a18d9fd8f48c338e092cd4a2d227b927b

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
Openstack Gerrit (openstack-gerrit) wrote : Fix merged to nova (stable/diablo)

Reviewed: https://review.openstack.org/2181
Committed: http://github.com/openstack/nova/commit/e9f427fc2e8014eeba3f46284c38414209fb0b7e
Submitter: Jenkins
Branch: stable/diablo

 tag in-stable-diablo
 done

commit e9f427fc2e8014eeba3f46284c38414209fb0b7e
Author: Vishvananda Ishaya <email address hidden>
Date: Wed Nov 16 10:17:23 2011 -0800

    Makes sure gateways forward properly

     * Fixes bug 890195
     * Fixes missing context in dhcp call
     * Adds test to verify call is correct

    (cherry picked from commit 41af372219793556e6ba335d765761fa277107df)

    Change-Id: Ic099082a18d9fd8f48c338e092cd4a2d227b927b

Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: essex-2 → 2012.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.