iptables-restore failed

[Christian Berendt]

I tried to spawn 10 instances (using two nova-compute nodes) and got the following error for 1 instance. In front of the error I saw a lot of calls to iptables-save. I guess iptables-restore is called to fast after the last iptables-save (appr. 15ms later).

[...]
2011-02-21 07:58:11,340 INFO nova.root [-] new_filter: # Generated by iptables-save v1.4.6 on Mon Feb 21 07:58:11 2011
[...]
2011-02-21 08:10:17,011 INFO nova.root [-] new_filter: # Generated by iptables-save v1.4.6 on Mon Feb 21 08:10:16 2011
[...]
2011-02-21 08:10:17,623 INFO nova.root [-] new_filter: # Generated by iptables-save v1.4.6 on Mon Feb 21 08:10:17 2011
[...]
2011-02-21 08:10:19,403 INFO nova.root [-] new_filter: # Generated by iptables-save v1.4.6 on Mon Feb 21 08:10:18 2011
[...]
2011-02-21 08:10:19,868 INFO nova.root [-] new_filter: # Generated by iptables-save v1.4.6 on Mon Feb 21 08:10:18 2011
[...]
2011-02-21 08:10:20,339 INFO nova.root [-] new_filter: # Generated by iptables-save v1.4.6 on Mon Feb 21 08:10:18 2011

2011-02-21 08:10:20,354 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib64/python2.6/site-packages/nova/exception.py", line 116, in _wrap
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt_conn.py", line 408, in spawn
(nova.exception): TRACE: self.firewall_driver.prepare_instance_filter(instance)
(nova.exception): TRACE: File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt_conn.py", line 1237, in prepare_instance_filter
(nova.exception): TRACE: self.apply_ruleset()
(nova.exception): TRACE: File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt_conn.py", line 1244, in apply_ruleset
(nova.exception): TRACE: process_input='\n'.join(new_filter))
(nova.exception): TRACE: File "/usr/lib64/python2.6/site-packages/nova/utils.py", line 149, in execute
(nova.exception): TRACE: cmd=cmd)
(nova.exception): TRACE: ProcessExecutionError: Unexpected error while running command.
(nova.exception): TRACE: Command: sudo iptables-restore
(nova.exception): TRACE: Exit code: 1
(nova.exception): TRACE: Stdout: ''
(nova.exception): TRACE: Stderr: 'iptables-restore: line 63 failed\n'
(nova.exception): TRACE:
2011-02-21 08:10:20,403 ERROR nova.compute.manager [BPEOLIU1NEACKYYV2QW2 berendt testing] instance 42: Failed to spawn
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 213, in run_instance
(nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
(nova.compute.manager): TRACE: File "/usr/lib64/python2.6/site-packages/nova/exception.py", line 122, in _wrap
(nova.compute.manager): TRACE: raise Error(str(e))
(nova.compute.manager): TRACE: Error: Unexpected error while running command.
(nova.compute.manager): TRACE: Command: sudo iptables-restore
(nova.compute.manager): TRACE: Exit code: 1
(nova.compute.manager): TRACE: Stdout: ''
(nova.compute.manager): TRACE: Stderr: 'iptables-restore: line 63 failed\n'
(nova.compute.manager): TRACE:

INSTANCE i-0000002a ami-2a1izx00 192.168.3.10 192.168.3.10 failed to spawn testing (testing, ares) 6 m1.tiny 2011-02-21 07:10:16 nova

[Soren Hansen (soren)]

Thanks for the bug report. I'm reworking our iptables handing to address this exact issue. I expect to finish this today.