Bug in logging potentionally causing neutron to fail

[Michal Arbet]

Hi,

Due to some completely different issue, we had ports with port_tags None created in production. Under normal circumstances, it would take some error to the log and go further.
But only because there is a "new_vlan_tags must be string" in that log, it threw out typeError and dropped the execution of code and the migration ended by code 500.

And without a message from the log, which is extremely bad.
It was able to go through the repair of the logo

2020-10-20 11:20:37.457 6 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-b8df2b85-d523-4217-b62c-e17b4404cade - - - - -] Error formatting log line msg="Port '*(port_name)s' has lost its vlan tag '*(vlan_tag)d'! Current vlan tag on this port is '*(new_vlan_tag)d'." err=TypeError('*d format: a number is required, not list')

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/762515

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Michal:

Those VLAN tags are an internal Neutron OVS agent configuration, to segregate the traffic from different networks. When or how can this situation happen?

Regards.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/762515
Committed: https://opendev.org/openstack/neutron/commit/7cf52bb5325ac73b7afa3212cb84d29831e3724a
Submitter: Zuul
Branch: master

commit 7cf52bb5325ac73b7afa3212cb84d29831e3724a
Author: Michal Arbet <email address hidden>
Date: Thu Nov 12 13:43:24 2020 +0100

    Fix migration failed due TypeError of new_vlan_tag

    There could be a situation, when there are ports
    with port_tags None created in OS infra.

    Under normal circumstances, it would take some error
    to the log and go further.
    But only because there is a "new_vlan_tags must be string"
    in that log, it is throwing out typeError and dropping
    the execution of code and the migration is ending by code 500.

    And this is all done without a message from
    the log, which is extremely bad.

    This patch is fixing this behaviour, and flow
    goes on.

    Change-Id: Id4ba8d32e6b2301f25c12f83a0708b62aa5b9d6d
    Closes-Bug: #1903989

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/787843

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/787843
Committed: https://opendev.org/openstack/neutron/commit/87c375bab79226a8df1e306f32fa29188d4d0ea5
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 87c375bab79226a8df1e306f32fa29188d4d0ea5
Author: Michal Arbet <email address hidden>
Date: Thu Nov 12 13:43:24 2020 +0100

    Fix migration failed due TypeError of new_vlan_tag

    There could be a situation, when there are ports
    with port_tags None created in OS infra.

    Under normal circumstances, it would take some error
    to the log and go further.
    But only because there is a "new_vlan_tags must be string"
    in that log, it is throwing out typeError and dropping
    the execution of code and the migration is ending by code 500.

    And this is all done without a message from
    the log, which is extremely bad.

    This patch is fixing this behaviour, and flow
    goes on.

    Change-Id: Id4ba8d32e6b2301f25c12f83a0708b62aa5b9d6d
    Closes-Bug: #1903989
    (cherry picked from commit 7cf52bb5325ac73b7afa3212cb84d29831e3724a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.1.2 release.

[Maximilian Stinsky (mstinsky)]

We were also hitting this after upgrading from rocky to stein und removed the log line via a manual backport. After testing the upgrade to train we can see that we still are hitting this same issue.

Are there any backports planned to earlier releases?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/853750

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/853751

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/neutron/+/853752

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/c/openstack/neutron/+/853753

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/c/openstack/neutron/+/853754

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/853750
Committed: https://opendev.org/openstack/neutron/commit/23ae2fa3bea86c21221451cab57e7c2fb8230332
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 23ae2fa3bea86c21221451cab57e7c2fb8230332
Author: Michal Arbet <email address hidden>
Date: Thu Nov 12 13:43:24 2020 +0100

    Fix migration failed due TypeError of new_vlan_tag

    There could be a situation, when there are ports
    with port_tags None created in OS infra.

    Under normal circumstances, it would take some error
    to the log and go further.
    But only because there is a "new_vlan_tags must be string"
    in that log, it is throwing out typeError and dropping
    the execution of code and the migration is ending by code 500.

    And this is all done without a message from
    the log, which is extremely bad.

    This patch is fixing this behaviour, and flow
    goes on.

    Change-Id: Id4ba8d32e6b2301f25c12f83a0708b62aa5b9d6d
    Closes-Bug: #1903989
    (cherry picked from commit 7cf52bb5325ac73b7afa3212cb84d29831e3724a)
    (cherry picked from commit 87c375bab79226a8df1e306f32fa29188d4d0ea5)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/853751
Committed: https://opendev.org/openstack/neutron/commit/754c34148e8619675b379a2e4cb37f226b65b499
Submitter: "Zuul (22348)"
Branch: stable/train

commit 754c34148e8619675b379a2e4cb37f226b65b499
Author: Michal Arbet <email address hidden>
Date: Thu Nov 12 13:43:24 2020 +0100

    Fix migration failed due TypeError of new_vlan_tag

    There could be a situation, when there are ports
    with port_tags None created in OS infra.

    Under normal circumstances, it would take some error
    to the log and go further.
    But only because there is a "new_vlan_tags must be string"
    in that log, it is throwing out typeError and dropping
    the execution of code and the migration is ending by code 500.

    And this is all done without a message from
    the log, which is extremely bad.

    This patch is fixing this behaviour, and flow
    goes on.

    Change-Id: Id4ba8d32e6b2301f25c12f83a0708b62aa5b9d6d
    Closes-Bug: #1903989
    (cherry picked from commit 7cf52bb5325ac73b7afa3212cb84d29831e3724a)
    (cherry picked from commit 87c375bab79226a8df1e306f32fa29188d4d0ea5)

[OpenStack Infra (hudson-openstack)]

Change abandoned by "liuyulong <email address hidden>" on branch: stable/queens
Review: https://review.opendev.org/c/openstack/neutron/+/853754

[OpenStack Infra (hudson-openstack)]

Change abandoned by "liuyulong <email address hidden>" on branch: stable/stein
Review: https://review.opendev.org/c/openstack/neutron/+/853752

[OpenStack Infra (hudson-openstack)]

Change abandoned by "liuyulong <email address hidden>" on branch: stable/rocky
Review: https://review.opendev.org/c/openstack/neutron/+/853753

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron train-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ussuri-eol release.