iptables is failing when lauching instances

Bug #701278 reported by Wayne A. Walls
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Soren Hansen

Bug Description

Using the packages from trunk, when I try to launch a new instance (currently using UEC Maverick 10.10 image), I get the following error in nova-compute.log

#####
root@wayne01:~# euca-describe-instances
RESERVATION r-ebeqxq2b dubproj
INSTANCE i-1 ami-iihfbgaq 192.168.0.3 192.168.0.3 pending mykey (dubproj, wayne01) 0 m1.tiny 2011-01-10 17:08:13
root@wayne01:~# tail -100 /var/log/nova/nova-compute.log
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] report_interval : 10 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] scheduler_manager : nova.scheduler.manager.SchedulerManager from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] helpshort : None from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] sql_connection : mysql://root:nova@10.127.35.119/nova from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] auth_token_ttl : 3600 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] rabbit_port : 5672 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] host : wayne01 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] volume_manager : nova.volume.manager.VolumeManager from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] rabbit_max_retries : 12 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] vpn_image_id : ami-cloudpipe from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] logfile : None from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] db_backend : sqlalchemy from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] compute_manager : nova.compute.manager.ComputeManager from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] rabbit_virtual_host : / from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] sql_retry_interval : 10 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] state_path : /var/lib/nova from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] sql_idle_timeout : 3600 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] glance_host : 127.0.0.1 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] default_image : ami-11111 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] control_exchange : nova from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] aws_secret_access_key : admin from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] s3_host : 10.127.35.119 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] periodic_interval : 60 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] enable_new_services : True from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] fake_network : False from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] network_topic : network from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.virt.libvirt_conn 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] Connecting to libvirt: qemu:///system from MainProcess (pid=5925) _get_connection /usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py:144
(nova.compute.manager 2011.1-LOCALBRANCH:LOCALREVISION): AUDIT [E8L0FA5W3465LYXOJANG dub dubproj] instance 1: starting...
(nova.virt.libvirt_conn 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] instance instance-00000001: starting toXML method from MainProcess (pid=5925) to_xml /usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py:544
(nova.virt.libvirt_conn 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] instance instance-00000001: starting toXML method from MainProcess (pid=5925) to_xml /usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py:547
(nova.virt.libvirt_conn 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] instance instance-00000001: finished toXML method from MainProcess (pid=5925) to_xml /usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py:590
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): INFO [N/A] called setup_basic_filtering in nwfilter
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): INFO [N/A] ensuring static filters
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): INFO [N/A] <nova.db.sqlalchemy.models.SecurityGroupIngressRule object at 0x3f79b50>
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): INFO [N/A] <nova.db.sqlalchemy.models.SecurityGroupIngressRule object at 0x3f793d0>
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): INFO [N/A] new_filter: # Generated by iptables-save v1.4.4 on Mon Jan 10 11:08:16 2011
*filter
:INPUT ACCEPT [83340:65595339]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [72152:31420948]
:nova-ipv4-fallback - [0:0]
:nova-local - [0:0]
:nova-inst-1 - [0:0]
:nova-sg-1 - [0:0]
-A nova-ipv4-fallback -j DROP
-A FORWARD -j nova-local
-A nova-local -d 192.168.0.3 -j nova-inst-1
-A nova-inst-1 -m state --state INVALID -j DROP
-A nova-inst-1 -m state --state ESTABLISHED,RELATED -j ACCEPT
-A nova-inst-1 -j nova-sg-1
-A nova-inst-1 -s 192.168.0.1 -p udp --sport 67 --dport 68
-A nova-inst-1 -j nova-ipv4-fallback
-A nova-sg-1 -p icmp -s 0.0.0.0/0 -m icmp --icmp_type -1/-1 -j ACCEPT
-A nova-sg-1 -p tcp -s 0.0.0.0/0 --dport 22 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A FORWARD -o br100 -j ACCEPT
-A FORWARD -i br100 -j ACCEPT
-A FORWARD -d 192.168.0.2/32 -p udp -m udp --dport 1194 -j ACCEPT
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
COMMIT
# Completed on Mon Jan 10 11:08:16 2011

(nova.exception 2011.1-LOCALBRANCH:LOCALREVISION): ERROR [N/A] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 82, in _wrap
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 367, in spawn
(nova.exception): TRACE: self.firewall_driver.prepare_instance_filter(instance)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 1009, in prepare_instance_filter
(nova.exception): TRACE: self.apply_ruleset()
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 1016, in apply_ruleset
(nova.exception): TRACE: process_input='\n'.join(new_filter))
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/utils.py", line 144, in execute
(nova.exception): TRACE: cmd=cmd)
(nova.exception): TRACE: ProcessExecutionError: Unexpected error while running command.
(nova.exception): TRACE: Command: sudo iptables-restore
(nova.exception): TRACE: Exit code: 2
(nova.exception): TRACE: Stdout: ''
(nova.exception): TRACE: Stderr: "iptables-restore v1.4.4: unknown option `--icmp_type'\nError occurred at line: 18\nTry `iptables-restore -h' or 'iptables-restore --help' for more information.\n"
(nova.exception): TRACE:
(nova.compute.manager 2011.1-LOCALBRANCH:LOCALREVISION): ERROR [E8L0FA5W3465LYXOJANG dub dubproj] instance 1: Failed to spawn
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 191, in run_instance
(nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 88, in _wrap
(nova.compute.manager): TRACE: raise Error(str(e))
(nova.compute.manager): TRACE: Error: Unexpected error while running command.
(nova.compute.manager): TRACE: Command: sudo iptables-restore
(nova.compute.manager): TRACE: Exit code: 2
(nova.compute.manager): TRACE: Stdout: ''
(nova.compute.manager): TRACE: Stderr: "iptables-restore v1.4.4: unknown option `--icmp_type'\nError occurred at line: 18\nTry `iptables-restore -h' or 'iptables-restore --help' for more information.\n"
(nova.compute.manager): TRACE:
libvir: QEMU error : Domain not found: no domain with matching name 'instance-00000001'

#####

I am seeing this behavior on 10.04, and 10.10. Here is the information on the iptables versions:

10.04
-----
root@wayne01:~# dpkg -l | grep iptables
ii iptables 1.4.4-2ubuntu2 administration tools for packet filtering an

10.10
-----
root@wayne07:~# dpkg -l | grep iptables
ii iptables 1.4.4-2ubuntu3 administration tools for packet filtering an

#####

I tried to upgrade to iptables 1.4.10-1ubuntu1, with one additional dependency (libnfnetlink0), and the issue is still valid.

#####

Cheers

Related branches

Revision history for this message
Soren Hansen (soren) wrote :

Perhaps iptables-restore doesn't automatically load modules like iptables does. We're certainly not using any exotic netfilter modules, and I've tested this on Ubuntu 10.10.

Changed in nova:
assignee: nobody → Soren Hansen (soren)
Revision history for this message
Soren Hansen (soren) wrote :

Nope, that's not it at all.

Changed in nova:
status: New → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → 2011.1
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.