OpenStack Compute (nova)

multi-process metadata server runs iptables setup multiple times

Bug #1097999 reported by Joe Gordon on 2013-01-10

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	Low	Anton	OpenStack Compute (nova) 2014.1 "icehouse"

Bug Description

Using Devstack (latest master 1/9/2012)

If I enable multi process metadata service (metadata_workers=5 in nova.conf)

the iptables are modified multiple times:
2013-01-10 00:25:16.0 25535 INFO nova.wsgi [-] metadata listening on 0.0.0.0:8775
2013-01-10 00:25:16.1 25535 INFO nova.service [-] Starting 5 workers
2013-01-10 00:25:16.3 25535 INFO nova.service [-] Started child 255442013-01-10 00:25:16.6 25535 INFO nova.service [-] Started child 255452013-01-10 00:25:16.8 25535 INFO nova.service [-] Started child 25546
2013-01-10 00:25:16.9 25544 DEBUG nova.openstack.common.lockutils [-] Got semaphore "iptables" for method "_apply"... inner /opt/stack/nova/nova/openstack/common/lockutils.py:185
2013-01-10 00:25:16.10 25544 DEBUG nova.openstack.common.lockutils [-] Attempting to grab file lock "iptables" for method "_apply"... inner /opt/stack/nova/nova/openstack/common/lockutils.py:189
2013-01-10 00:25:16.11 25544 DEBUG nova.openstack.common.lockutils [-] Got file lock "iptables" at /opt/stack/data/nova/nova-iptables for method "_apply"... inner /opt/stack/nova/nova/openstack/common/lockutils.py:219
2013-01-10 00:25:16.11 25535 INFO nova.service [-] Started child 25547
2013-01-10 00:25:16.18 25544 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t filter execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:16.20 25535 INFO nova.service [-] Started child 25548
2013-01-10 00:25:16.16 25546 DEBUG nova.openstack.common.lockutils [-] Got semaphore "iptables" for method "_apply"... inner /opt/stack/nova/nova/openstack/common/lockutils.py:185
2013-01-10 00:25:16.22 25546 DEBUG nova.openstack.common.lockutils [-] Attempting to grab file lock "iptables" for method "_apply"... inner /opt/stack/nova/nova/openstack/common/lockutils.py:189
2013-01-10 00:25:16.27 25548 DEBUG nova.openstack.common.lockutils [-] Got semaphore "iptables" for method "_apply"... inner /opt/stack/nova/nova/openstack/common/lockutils.py:185
2013-01-10 00:25:16.17 25545 DEBUG nova.openstack.common.lockutils [-] Got semaphore "iptables" for method "_apply"... inner /opt/stack/nova/nova/openstack/common/lockutils.py:185
2013-01-10 00:25:16.33 25545 DEBUG nova.openstack.common.lockutils [-] Attempting to grab file lock "iptables" for method "_apply"... inner /opt/stack/nova/nova/openstack/common/lockutils.py:189
2013-01-10 00:25:16.32 25547 DEBUG nova.openstack.common.lockutils [-] Got semaphore "iptables" for method "_apply"... inner /opt/stack/nova/nova/openstack/common/lockutils.py:185
2013-01-10 00:25:16.33 25547 DEBUG nova.openstack.common.lockutils [-] Attempting to grab file lock "iptables" for method "_apply"... inner /opt/stack/nova/nova/openstack/common/lockutils.py:189
2013-01-10 00:25:16.42 25548 DEBUG nova.openstack.common.lockutils [-] Attempting to grab file lock "iptables" for method "_apply"... inner /opt/stack/nova/nova/openstack/common/lockutils.py:189
2013-01-10 00:25:16.214 25544 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:16.216 25544 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:16.368 25544 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:16.370 25544 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t mangle execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:16.535 25544 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:16.536 25544 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:16.703 25544 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:16.704 25544 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t nat execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:16.855 25544 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:16.856 25544 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:17.18 25544 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:17.19 25544 DEBUG nova.network.linux_net [-] IPTablesManager.apply completed with success _apply /opt/stack/nova/nova/network/linux_net.py:385
2013-01-10 00:25:17.20 25544 INFO nova.metadata.wsgi.server [-] (25544) wsgi starting up on http://0.0.0.0:8775/

2013-01-10 00:25:17.25 25548 DEBUG nova.openstack.common.lockutils [-] Got file lock "iptables" at /opt/stack/data/nova/nova-iptables for method "_apply"... inner /opt/stack/nova/nova/openstack/common/lockutils.py:219
2013-01-10 00:25:17.26 25548 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t filter execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:17.180 25548 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:17.182 25548 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:17.340 25548 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:17.342 25548 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t mangle execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:17.510 25548 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:17.511 25548 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:17.666 25548 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:17.667 25548 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t nat execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:17.800 25548 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:17.801 25548 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:17.952 25548 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:17.953 25548 DEBUG nova.network.linux_net [-] IPTablesManager.apply completed with success _apply /opt/stack/nova/nova/network/linux_net.py:385
2013-01-10 00:25:17.955 25548 INFO nova.metadata.wsgi.server [-] (25548) wsgi starting up on http://0.0.0.0:8775/

2013-01-10 00:25:17.962 25546 DEBUG nova.openstack.common.lockutils [-] Got file lock "iptables" at /opt/stack/data/nova/nova-iptables for method "_apply"... inner /opt/stack/nova/nova/openstack/common/lockutils.py:219
2013-01-10 00:25:17.964 25546 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t filter execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:18.120 25546 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:18.121 25546 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:18.278 25546 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:18.279 25546 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t mangle execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:18.439 25546 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:18.441 25546 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:18.596 25546 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:18.598 25546 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t nat execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:18.750 25546 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:18.751 25546 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:18.932 25546 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:18.932 25546 DEBUG nova.network.linux_net [-] IPTablesManager.apply completed with success _apply /opt/stack/nova/nova/network/linux_net.py:385
2013-01-10 00:25:18.934 25546 INFO nova.metadata.wsgi.server [-] (25546) wsgi starting up on http://0.0.0.0:8775/

2013-01-10 00:25:18.941 25545 DEBUG nova.openstack.common.lockutils [-] Got file lock "iptables" at /opt/stack/data/nova/nova-iptables for method "_apply"... inner /opt/stack/nova/nova/openstack/common/lockutils.py:219
2013-01-10 00:25:18.942 25545 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t filter execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:19.94 25545 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:19.95 25545 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:19.210 25545 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:19.210 25545 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t mangle execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:19.328 25545 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:19.330 25545 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:19.470 25545 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:19.471 25545 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t nat execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:19.617 25545 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:19.619 25545 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:19.735 25545 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:19.736 25545 DEBUG nova.network.linux_net [-] IPTablesManager.apply completed with success _apply /opt/stack/nova/nova/network/linux_net.py:385
2013-01-10 00:25:19.737 25547 DEBUG nova.openstack.common.lockutils [-] Got file lock "iptables" at /opt/stack/data/nova/nova-iptables for method "_apply"... inner /opt/stack/nova/nova/openstack/common/lockutils.py:219
2013-01-10 00:25:19.738 25547 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t filter execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:19.739 25545 INFO nova.metadata.wsgi.server [-] (25545) wsgi starting up on http://0.0.0.0:8775/

2013-01-10 00:25:19.854 25547 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:19.855 25547 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:19.971 25547 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:19.972 25547 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t mangle execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:20.88 25547 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:20.89 25547 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:20.205 25547 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:20.206 25547 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t nat execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:20.327 25547 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:20.328 25547 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c execute /opt/stack/nova/nova/utils.py:202
2013-01-10 00:25:20.447 25547 DEBUG nova.utils [-] Result was 0 execute /opt/stack/nova/nova/utils.py:226
2013-01-10 00:25:20.447 25547 DEBUG nova.network.linux_net [-] IPTablesManager.apply completed with success _apply /opt/stack/nova/nova/network/linux_net.py:385
2013-01-10 00:25:20.449 25547 INFO nova.metadata.wsgi.server [-] (25547) wsgi starting up on http://0.0.0.0:8775/

Vish Ishaya (vishvananda) on 2013-02-26

Changed in nova:
status:	New → Triaged
importance:	Undecided → Low

Anton (agorenkov) on 2013-12-11

Changed in nova:
assignee:	nobody → Anton (agorenkov)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-12-16: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/62402

Changed in nova:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-12-23: Fix merged to nova (master)

Reviewed: https://review.openstack.org/62402
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5161d6c0023151d39fb56a85f739063205e676f4
Submitter: Jenkins
Branch: master

commit 5161d6c0023151d39fb56a85f739063205e676f4
Author: Anton Gorenkov <email address hidden>
Date: Mon Dec 16 17:37:50 2013 +0200

Initialize iptables rules on initialization of MetadataManager

    To avoid multiple initialization of iptables rules if there are a few
    workers for metadata service, perform iptables configuration in
    __init__() of MetadataManager.

Change-Id: I674c04f973318f06cbb98693f0a884c824af8748
Closes-Bug: #1097999

Changed in nova:
status:	In Progress → Fix Committed

Russell Bryant (russellb) on 2014-01-13

Changed in nova:
milestone:	none → icehouse-2

Thierry Carrez (ttx) on 2014-01-22

Changed in nova:
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2014-04-17

Changed in nova:
milestone:	icehouse-2 → 2014.1

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.