neutron

[OVN] OVN metadata "MetadataProxyHandler" not working if workers=0

Bug #1993181 reported by Rodolfo Alonso
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Low
Rodolfo Alonso

Bug Description

The OVN metadata service can spawn several "MetadataProxyHandler" instances in separate processes. The number of workers ("metadata_workers") will define how many processes will be created.

If this configuration variable is set to zero, the parent process (in the case the OVN metadata agent) won't create a new process but start the service. The "wsgi.Server" instance will be started instead; that means the "wsgi.Server" instance will use the thread pool "self.pool = eventlet.GreenPool(1)" to execute the application. The server application is an instance of "neutron.agent.ovn.metadata.server.MetadataProxyHandler".

This instance of "MetadataProxyHandler" has a SB IDL connection. This IDL connection is initialized in the "post_fork_initialize" call, that is executed when the event (resources.PROCESS, events.AFTER_INIT) is received. The problem is that when the "MetadataProxyHandler" instance is called inside a thread, not a process, this method is not called and the IDL connection is not initialized.

In order to solve this issue and at the same time save OVN DB connections, the proposal is to resuse the OVN metadata IDL connection with the Proxy thread.

Rodolfo Alonso (rodolfo-alonso-hernandez)
Changed in neutron:
importance: Undecided → Low
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)
Oleg Bondarev (obondarev)
tags: added: ovn
Changed in neutron:
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/861649

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/861649
Committed: https://opendev.org/openstack/neutron/commit/f43891bf866b65ceef0e51633afbbf57ee2a6be8
Submitter: "Zuul (22348)"
Branch: master

commit f43891bf866b65ceef0e51633afbbf57ee2a6be8
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Oct 5 13:22:29 2022 +0200

    [OVN] Allow to execute ``MetadataProxyHandler`` in a local thread

    If configuration option "metadata_workers=0", the OVN metadata agent
    will try to spawn the ``MetadataProxyHandler`` instance inside a local
    thread, instead of creating a new process. In this case, the method
    ``MetadataProxyHandler.post_fork_initialize`` is never called and the
    SB IDL is never created.

    This patch passes the OVN metadata agent SB IDL instance to the proxy
    handler instance. This also reduces the number of OVN database active
    connections.

    Closes-Bug: #1993181
    Change-Id: If9d827228002de7e3a55be660da266b60b0dfb79

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/861733

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/861734

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/861735

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/861736

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/861751

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/861733
Committed: https://opendev.org/openstack/neutron/commit/753e7ceb2d04f6e723613407f0703b9221c5da1e
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 753e7ceb2d04f6e723613407f0703b9221c5da1e
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Oct 5 13:22:29 2022 +0200

    [OVN] Allow to execute ``MetadataProxyHandler`` in a local thread

    If configuration option "metadata_workers=0", the OVN metadata agent
    will try to spawn the ``MetadataProxyHandler`` instance inside a local
    thread, instead of creating a new process. In this case, the method
    ``MetadataProxyHandler.post_fork_initialize`` is never called and the
    SB IDL is never created.

    This patch passes the OVN metadata agent SB IDL instance to the proxy
    handler instance. This also reduces the number of OVN database active
    connections.

    Closes-Bug: #1993181
    Change-Id: If9d827228002de7e3a55be660da266b60b0dfb79
    (cherry picked from commit f43891bf866b65ceef0e51633afbbf57ee2a6be8)

tags: added: in-stable-zed
tags: added: in-stable-yoga
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/861734
Committed: https://opendev.org/openstack/neutron/commit/c8409a33d9d3e7dc3f83bf36053781cf9a5345b1
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit c8409a33d9d3e7dc3f83bf36053781cf9a5345b1
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Oct 5 13:22:29 2022 +0200

    [OVN] Allow to execute ``MetadataProxyHandler`` in a local thread

    If configuration option "metadata_workers=0", the OVN metadata agent
    will try to spawn the ``MetadataProxyHandler`` instance inside a local
    thread, instead of creating a new process. In this case, the method
    ``MetadataProxyHandler.post_fork_initialize`` is never called and the
    SB IDL is never created.

    This patch passes the OVN metadata agent SB IDL instance to the proxy
    handler instance. This also reduces the number of OVN database active
    connections.

    Closes-Bug: #1993181
    Change-Id: If9d827228002de7e3a55be660da266b60b0dfb79
    (cherry picked from commit f43891bf866b65ceef0e51633afbbf57ee2a6be8)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/861735
Committed: https://opendev.org/openstack/neutron/commit/560ea25085720c5890e34ddf8cd9f3320fd9d583
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 560ea25085720c5890e34ddf8cd9f3320fd9d583
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Oct 5 13:22:29 2022 +0200

    [OVN] Allow to execute ``MetadataProxyHandler`` in a local thread

    If configuration option "metadata_workers=0", the OVN metadata agent
    will try to spawn the ``MetadataProxyHandler`` instance inside a local
    thread, instead of creating a new process. In this case, the method
    ``MetadataProxyHandler.post_fork_initialize`` is never called and the
    SB IDL is never created.

    This patch passes the OVN metadata agent SB IDL instance to the proxy
    handler instance. This also reduces the number of OVN database active
    connections.

    Closes-Bug: #1993181
    Change-Id: If9d827228002de7e3a55be660da266b60b0dfb79
    (cherry picked from commit f43891bf866b65ceef0e51633afbbf57ee2a6be8)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/861736
Committed: https://opendev.org/openstack/neutron/commit/14190a612f8d760abdaff2ba8559980354942c8a
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 14190a612f8d760abdaff2ba8559980354942c8a
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Oct 5 13:22:29 2022 +0200

    [OVN] Allow to execute ``MetadataProxyHandler`` in a local thread

    If configuration option "metadata_workers=0", the OVN metadata agent
    will try to spawn the ``MetadataProxyHandler`` instance inside a local
    thread, instead of creating a new process. In this case, the method
    ``MetadataProxyHandler.post_fork_initialize`` is never called and the
    SB IDL is never created.

    This patch passes the OVN metadata agent SB IDL instance to the proxy
    handler instance. This also reduces the number of OVN database active
    connections.

    Closes-Bug: #1993181
    Change-Id: If9d827228002de7e3a55be660da266b60b0dfb79
    (cherry picked from commit f43891bf866b65ceef0e51633afbbf57ee2a6be8)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/861751
Committed: https://opendev.org/openstack/neutron/commit/70a2029ca1309ef4a784d2f08f5b5f4bc69617a6
Submitter: "Zuul (22348)"
Branch: master

commit 70a2029ca1309ef4a784d2f08f5b5f4bc69617a6
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Oct 5 21:35:48 2022 +0200

    [OVN] Set the default OVN metadata worker number to 0

    If the configuration variable ``metadata_workers`` is not set,
    the default value for ML2/OVN mechanism driver will be zero.
    This value defines the number of separate workers that will
    be spawned by the OVN metadata agent. By default, the OVN
    metadata proxy handler will be executed inside the same OVN
    metadata agent process, in a parallel thread. That reduces the
    number of OVN SB database connections to one.

    Related-Bug: #1993181
    Change-Id: I8bc4f2784ccefe6078506bc27cc7a98c48192ad2

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/zed)

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/862589

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/yoga)

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/862710

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/xena)

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/862711

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/862712

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/862712
Committed: https://opendev.org/openstack/neutron/commit/93e7091a13f8471cfe77310802c12ede479e779f
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 93e7091a13f8471cfe77310802c12ede479e779f
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Oct 5 21:35:48 2022 +0200

    [OVN] Set the default OVN metadata worker number to 0

    If the configuration variable ``metadata_workers`` is not set,
    the default value for ML2/OVN mechanism driver will be zero.
    This value defines the number of separate workers that will
    be spawned by the OVN metadata agent. By default, the OVN
    metadata proxy handler will be executed inside the same OVN
    metadata agent process, in a parallel thread. That reduces the
    number of OVN SB database connections to one.

    Related-Bug: #1993181
    Change-Id: I8bc4f2784ccefe6078506bc27cc7a98c48192ad2
    (cherry picked from commit 70a2029ca1309ef4a784d2f08f5b5f4bc69617a6)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/862711
Committed: https://opendev.org/openstack/neutron/commit/40254f7235a03ad28a4793a85aea96866f1fad7a
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 40254f7235a03ad28a4793a85aea96866f1fad7a
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Oct 5 21:35:48 2022 +0200

    [OVN] Set the default OVN metadata worker number to 0

    If the configuration variable ``metadata_workers`` is not set,
    the default value for ML2/OVN mechanism driver will be zero.
    This value defines the number of separate workers that will
    be spawned by the OVN metadata agent. By default, the OVN
    metadata proxy handler will be executed inside the same OVN
    metadata agent process, in a parallel thread. That reduces the
    number of OVN SB database connections to one.

    Related-Bug: #1993181
    Change-Id: I8bc4f2784ccefe6078506bc27cc7a98c48192ad2
    (cherry picked from commit 70a2029ca1309ef4a784d2f08f5b5f4bc69617a6)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/862710
Committed: https://opendev.org/openstack/neutron/commit/56841381ef327aa940bd0a630ec0bf8d7066056c
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 56841381ef327aa940bd0a630ec0bf8d7066056c
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Oct 5 21:35:48 2022 +0200

    [OVN] Set the default OVN metadata worker number to 0

    If the configuration variable ``metadata_workers`` is not set,
    the default value for ML2/OVN mechanism driver will be zero.
    This value defines the number of separate workers that will
    be spawned by the OVN metadata agent. By default, the OVN
    metadata proxy handler will be executed inside the same OVN
    metadata agent process, in a parallel thread. That reduces the
    number of OVN SB database connections to one.

    Related-Bug: #1993181
    Change-Id: I8bc4f2784ccefe6078506bc27cc7a98c48192ad2
    (cherry picked from commit 70a2029ca1309ef4a784d2f08f5b5f4bc69617a6)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/862589
Committed: https://opendev.org/openstack/neutron/commit/8dd2dea65d3c90ba7be6907142b7ccbd05ea4238
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 8dd2dea65d3c90ba7be6907142b7ccbd05ea4238
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Oct 5 21:35:48 2022 +0200

    [OVN] Set the default OVN metadata worker number to 0

    If the configuration variable ``metadata_workers`` is not set,
    the default value for ML2/OVN mechanism driver will be zero.
    This value defines the number of separate workers that will
    be spawned by the OVN metadata agent. By default, the OVN
    metadata proxy handler will be executed inside the same OVN
    metadata agent process, in a parallel thread. That reduces the
    number of OVN SB database connections to one.

    Related-Bug: #1993181
    Change-Id: I8bc4f2784ccefe6078506bc27cc7a98c48192ad2
    (cherry picked from commit 70a2029ca1309ef4a784d2f08f5b5f4bc69617a6)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.6.0

This issue was fixed in the openstack/neutron 18.6.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 19.5.0

This issue was fixed in the openstack/neutron 19.5.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 22.0.0.0rc1

This issue was fixed in the openstack/neutron 22.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 20.3.0

This issue was fixed in the openstack/neutron 20.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 21.1.0

This issue was fixed in the openstack/neutron 21.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/901227

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/901227
Committed: https://opendev.org/openstack/neutron/commit/407585f99f82c0136f66f0b4874da58500574c63
Submitter: "Zuul (22348)"
Branch: master

commit 407585f99f82c0136f66f0b4874da58500574c63
Author: Bartosz Bezak <email address hidden>
Date: Fri Nov 17 10:33:32 2023 +0100

    docs: update default value of metadata workers for ml2/ovn

    [1] changed that value to 0

    [1] https://review.opendev.org/c/openstack/neutron/+/861751

    Related-Bug: #1993181
    Change-Id: I7009e8a9fa8a61cc796d9592db0cf68e07e5884d

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/2023.2)

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/neutron/+/901495

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/2023.1)

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/901496

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/zed)

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/901497

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/yoga)

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/901498

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/xena)

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/901499

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/901500

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/901495
Committed: https://opendev.org/openstack/neutron/commit/2200101f9ffd3c6e978d207a031aca84ec26ab07
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 2200101f9ffd3c6e978d207a031aca84ec26ab07
Author: Bartosz Bezak <email address hidden>
Date: Fri Nov 17 10:33:32 2023 +0100

    docs: update default value of metadata workers for ml2/ovn

    [1] changed that value to 0

    [1] https://review.opendev.org/c/openstack/neutron/+/861751

    Related-Bug: #1993181
    Change-Id: I7009e8a9fa8a61cc796d9592db0cf68e07e5884d
    (cherry picked from commit 407585f99f82c0136f66f0b4874da58500574c63)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/901496
Committed: https://opendev.org/openstack/neutron/commit/458915e56f4d83958f362478787b864ed71b9d15
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 458915e56f4d83958f362478787b864ed71b9d15
Author: Bartosz Bezak <email address hidden>
Date: Fri Nov 17 10:33:32 2023 +0100

    docs: update default value of metadata workers for ml2/ovn

    [1] changed that value to 0

    [1] https://review.opendev.org/c/openstack/neutron/+/861751

    Related-Bug: #1993181
    Change-Id: I7009e8a9fa8a61cc796d9592db0cf68e07e5884d
    (cherry picked from commit 407585f99f82c0136f66f0b4874da58500574c63)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/901497
Committed: https://opendev.org/openstack/neutron/commit/b2100c6a29a66f0dba9729f9570c2b90e87d6528
Submitter: "Zuul (22348)"
Branch: stable/zed

commit b2100c6a29a66f0dba9729f9570c2b90e87d6528
Author: Bartosz Bezak <email address hidden>
Date: Fri Nov 17 10:33:32 2023 +0100

    docs: update default value of metadata workers for ml2/ovn

    [1] changed that value to 0

    [1] https://review.opendev.org/c/openstack/neutron/+/861751

    Related-Bug: #1993181
    Change-Id: I7009e8a9fa8a61cc796d9592db0cf68e07e5884d
    (cherry picked from commit 407585f99f82c0136f66f0b4874da58500574c63)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/901498
Committed: https://opendev.org/openstack/neutron/commit/9776e9e74b8e06d9507e299c546ba06d721f7455
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 9776e9e74b8e06d9507e299c546ba06d721f7455
Author: Bartosz Bezak <email address hidden>
Date: Fri Nov 17 10:33:32 2023 +0100

    docs: update default value of metadata workers for ml2/ovn

    [1] changed that value to 0

    [1] https://review.opendev.org/c/openstack/neutron/+/861751

    Related-Bug: #1993181
    Change-Id: I7009e8a9fa8a61cc796d9592db0cf68e07e5884d
    (cherry picked from commit 407585f99f82c0136f66f0b4874da58500574c63)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/901500
Committed: https://opendev.org/openstack/neutron/commit/7a2332876f472d24a782c4c05afd8401b3a610c5
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 7a2332876f472d24a782c4c05afd8401b3a610c5
Author: Bartosz Bezak <email address hidden>
Date: Fri Nov 17 10:33:32 2023 +0100

    docs: update default value of metadata workers for ml2/ovn

    [1] changed that value to 0

    [1] https://review.opendev.org/c/openstack/neutron/+/861751

    Related-Bug: #1993181
    Change-Id: I7009e8a9fa8a61cc796d9592db0cf68e07e5884d
    (cherry picked from commit 407585f99f82c0136f66f0b4874da58500574c63)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/901499
Committed: https://opendev.org/openstack/neutron/commit/5086805dd1da3db14801c9115983f28786e9715b
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 5086805dd1da3db14801c9115983f28786e9715b
Author: Bartosz Bezak <email address hidden>
Date: Fri Nov 17 10:33:32 2023 +0100

    docs: update default value of metadata workers for ml2/ovn

    [1] changed that value to 0

    [1] https://review.opendev.org/c/openstack/neutron/+/861751

    Related-Bug: #1993181
    Change-Id: I7009e8a9fa8a61cc796d9592db0cf68e07e5884d
    (cherry picked from commit 407585f99f82c0136f66f0b4874da58500574c63)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.