Using the packages from trunk, when I try to launch a new instance (currently using UEC Maverick 10.10 image), I get the following error in nova-compute.log
#####
root@wayne01:~# euca-describe-instances
RESERVATION r-ebeqxq2b dubproj
INSTANCE i-1 ami-iihfbgaq 192.168.0.3 192.168.0.3 pending mykey (dubproj, wayne01) 0 m1.tiny 2011-01-10 17:08:13
root@wayne01:~# tail -100 /var/log/nova/nova-compute.log
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] report_interval : 10 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] scheduler_manager : nova.scheduler.manager.SchedulerManager from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] helpshort : None from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] sql_connection : mysql://root:nova@10.127.35.119/nova from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] auth_token_ttl : 3600 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] rabbit_port : 5672 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] host : wayne01 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] volume_manager : nova.volume.manager.VolumeManager from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] rabbit_max_retries : 12 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] vpn_image_id : ami-cloudpipe from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] logfile : None from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] db_backend : sqlalchemy from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] compute_manager : nova.compute.manager.ComputeManager from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] rabbit_virtual_host : / from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] sql_retry_interval : 10 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] state_path : /var/lib/nova from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] sql_idle_timeout : 3600 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] glance_host : 127.0.0.1 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] default_image : ami-11111 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] control_exchange : nova from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] aws_secret_access_key : admin from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] s3_host : 10.127.35.119 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] periodic_interval : 60 from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] enable_new_services : True from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] fake_network : False from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] network_topic : network from MainProcess (pid=5925) serve /usr/lib/pymodules/python2.6/nova/service.py:236
(nova.virt.libvirt_conn 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] Connecting to libvirt: qemu:///system from MainProcess (pid=5925) _get_connection /usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py:144
(nova.compute.manager 2011.1-LOCALBRANCH:LOCALREVISION): AUDIT [E8L0FA5W3465LYXOJANG dub dubproj] instance 1: starting...
(nova.virt.libvirt_conn 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] instance instance-00000001: starting toXML method from MainProcess (pid=5925) to_xml /usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py:544
(nova.virt.libvirt_conn 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] instance instance-00000001: starting toXML method from MainProcess (pid=5925) to_xml /usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py:547
(nova.virt.libvirt_conn 2011.1-LOCALBRANCH:LOCALREVISION): DEBUG [N/A] instance instance-00000001: finished toXML method from MainProcess (pid=5925) to_xml /usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py:590
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): INFO [N/A] called setup_basic_filtering in nwfilter
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): INFO [N/A] ensuring static filters
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): INFO [N/A] <nova.db.sqlalchemy.models.SecurityGroupIngressRule object at 0x3f79b50>
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): INFO [N/A] <nova.db.sqlalchemy.models.SecurityGroupIngressRule object at 0x3f793d0>
(nova.root 2011.1-LOCALBRANCH:LOCALREVISION): INFO [N/A] new_filter: # Generated by iptables-save v1.4.4 on Mon Jan 10 11:08:16 2011
*filter
:INPUT ACCEPT [83340:65595339]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [72152:31420948]
:nova-ipv4-fallback - [0:0]
:nova-local - [0:0]
:nova-inst-1 - [0:0]
:nova-sg-1 - [0:0]
-A nova-ipv4-fallback -j DROP
-A FORWARD -j nova-local
-A nova-local -d 192.168.0.3 -j nova-inst-1
-A nova-inst-1 -m state --state INVALID -j DROP
-A nova-inst-1 -m state --state ESTABLISHED,RELATED -j ACCEPT
-A nova-inst-1 -j nova-sg-1
-A nova-inst-1 -s 192.168.0.1 -p udp --sport 67 --dport 68
-A nova-inst-1 -j nova-ipv4-fallback
-A nova-sg-1 -p icmp -s 0.0.0.0/0 -m icmp --icmp_type -1/-1 -j ACCEPT
-A nova-sg-1 -p tcp -s 0.0.0.0/0 --dport 22 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A FORWARD -o br100 -j ACCEPT
-A FORWARD -i br100 -j ACCEPT
-A FORWARD -d 192.168.0.2/32 -p udp -m udp --dport 1194 -j ACCEPT
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
COMMIT
# Completed on Mon Jan 10 11:08:16 2011
(nova.exception 2011.1-LOCALBRANCH:LOCALREVISION): ERROR [N/A] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 82, in _wrap
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 367, in spawn
(nova.exception): TRACE: self.firewall_driver.prepare_instance_filter(instance)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 1009, in prepare_instance_filter
(nova.exception): TRACE: self.apply_ruleset()
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 1016, in apply_ruleset
(nova.exception): TRACE: process_input='\n'.join(new_filter))
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/utils.py", line 144, in execute
(nova.exception): TRACE: cmd=cmd)
(nova.exception): TRACE: ProcessExecutionError: Unexpected error while running command.
(nova.exception): TRACE: Command: sudo iptables-restore
(nova.exception): TRACE: Exit code: 2
(nova.exception): TRACE: Stdout: ''
(nova.exception): TRACE: Stderr: "iptables-restore v1.4.4: unknown option `--icmp_type'\nError occurred at line: 18\nTry `iptables-restore -h' or 'iptables-restore --help' for more information.\n"
(nova.exception): TRACE:
(nova.compute.manager 2011.1-LOCALBRANCH:LOCALREVISION): ERROR [E8L0FA5W3465LYXOJANG dub dubproj] instance 1: Failed to spawn
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 191, in run_instance
(nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 88, in _wrap
(nova.compute.manager): TRACE: raise Error(str(e))
(nova.compute.manager): TRACE: Error: Unexpected error while running command.
(nova.compute.manager): TRACE: Command: sudo iptables-restore
(nova.compute.manager): TRACE: Exit code: 2
(nova.compute.manager): TRACE: Stdout: ''
(nova.compute.manager): TRACE: Stderr: "iptables-restore v1.4.4: unknown option `--icmp_type'\nError occurred at line: 18\nTry `iptables-restore -h' or 'iptables-restore --help' for more information.\n"
(nova.compute.manager): TRACE:
libvir: QEMU error : Domain not found: no domain with matching name 'instance-00000001'
#####
I am seeing this behavior on 10.04, and 10.10. Here is the information on the iptables versions:
10.04
-----
root@wayne01:~# dpkg -l | grep iptables
ii iptables 1.4.4-2ubuntu2 administration tools for packet filtering an
10.10
-----
root@wayne07:~# dpkg -l | grep iptables
ii iptables 1.4.4-2ubuntu3 administration tools for packet filtering an
#####
I tried to upgrade to iptables 1.4.10-1ubuntu1, with one additional dependency (libnfnetlink0), and the issue is still valid.
#####
Cheers
Perhaps iptables-restore doesn't automatically load modules like iptables does. We're certainly not using any exotic netfilter modules, and I've tested this on Ubuntu 10.10.