OVN agent heartbeat timestamp format changed unexpectedly

[Pierre Riteau]

Following an upgrade from Wallaby to Yoga of a Neutron OVN deployment, I discovered that openstack-exporter would fail to return metrics for Neutron agent states with the following error:

time="2023-02-23T10:49:33Z" level=error msg="Failed to collect metric for exporter: neutron, error: failed to collect metric: agent_state, error: parsing time \"2023-02-23 10:48:55.729000+00:00\": extra text: \"+00:00\"" source="exporter.go:123"

I tracked it down to the following change which has also been backported to stable/branches (so a recent Wallaby would also be affected): https://review.opendev.org/c/openstack/neutron/+/844179

With this change, heartbeat timestamp includes timezone when previously it didn't, causing clients such as the gophercloud library (which openstack-exporter uses) to fail parsing of the response.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/874926

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/875485

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/875486

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/874926
Committed: https://opendev.org/openstack/neutron/commit/827fbd01c306ce292e46b0d881bc3cc804202285
Submitter: "Zuul (22348)"
Branch: master

commit 827fbd01c306ce292e46b0d881bc3cc804202285
Author: Pierre Riteau <email address hidden>
Date: Fri Feb 24 05:34:07 2023 +0100

    Normalise format of OVN agent heartbeat timestamp

    A recent change [1] to show the real heartbeat timestamp from OVN agents
    had a side effect of changing the timestamp format, which now includes a
    timezone:

        +-------------------+----------------------------------+
        | Field | Value |
        +-------------------+----------------------------------+
        | last_heartbeat_at | 2023-02-23 14:12:07.471000+00:00 |
        +-------------------+----------------------------------+

    This unexpected format change causes some clients to fail to parse the
    response to GET /v2.0/agents.

    Normalise the format of the timestamp to remove timezone information.
    Also remove the microsecond part, which was not done for OVN, but is
    absent from other network agents.

    [1] https://review.opendev.org/c/openstack/neutron/+/844179

    Closes-Bug: #2008257
    Change-Id: I75a37fb9b49a421e4524da6b56ef8362ceb6107b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/875485
Committed: https://opendev.org/openstack/neutron/commit/177956506e7cb9ec21676f0c9260c3d55c08a444
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 177956506e7cb9ec21676f0c9260c3d55c08a444
Author: Pierre Riteau <email address hidden>
Date: Fri Feb 24 05:34:07 2023 +0100

    Normalise format of OVN agent heartbeat timestamp

    A recent change [1] to show the real heartbeat timestamp from OVN agents
    had a side effect of changing the timestamp format, which now includes a
    timezone:

        +-------------------+----------------------------------+
        | Field | Value |
        +-------------------+----------------------------------+
        | last_heartbeat_at | 2023-02-23 14:12:07.471000+00:00 |
        +-------------------+----------------------------------+

    This unexpected format change causes some clients to fail to parse the
    response to GET /v2.0/agents.

    Normalise the format of the timestamp to remove timezone information.
    Also remove the microsecond part, which was not done for OVN, but is
    absent from other network agents.

    [1] https://review.opendev.org/c/openstack/neutron/+/844179

    Closes-Bug: #2008257
    Change-Id: I75a37fb9b49a421e4524da6b56ef8362ceb6107b
    (cherry picked from commit 827fbd01c306ce292e46b0d881bc3cc804202285)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/875486
Committed: https://opendev.org/openstack/neutron/commit/1d611f4a7e1d6d1e9103845390e4bea0bd4c6a19
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 1d611f4a7e1d6d1e9103845390e4bea0bd4c6a19
Author: Pierre Riteau <email address hidden>
Date: Fri Feb 24 05:34:07 2023 +0100

    Normalise format of OVN agent heartbeat timestamp

    A recent change [1] to show the real heartbeat timestamp from OVN agents
    had a side effect of changing the timestamp format, which now includes a
    timezone:

        +-------------------+----------------------------------+
        | Field | Value |
        +-------------------+----------------------------------+
        | last_heartbeat_at | 2023-02-23 14:12:07.471000+00:00 |
        +-------------------+----------------------------------+

    This unexpected format change causes some clients to fail to parse the
    response to GET /v2.0/agents.

    Normalise the format of the timestamp to remove timezone information.
    Also remove the microsecond part, which was not done for OVN, but is
    absent from other network agents.

    [1] https://review.opendev.org/c/openstack/neutron/+/844179

    Closes-Bug: #2008257
    Change-Id: I75a37fb9b49a421e4524da6b56ef8362ceb6107b
    (cherry picked from commit 827fbd01c306ce292e46b0d881bc3cc804202285)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/875682

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/875683

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/875683
Committed: https://opendev.org/openstack/neutron/commit/f285dc23515811433fe3f3c65214f706541edbc7
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit f285dc23515811433fe3f3c65214f706541edbc7
Author: Pierre Riteau <email address hidden>
Date: Fri Feb 24 05:34:07 2023 +0100

    Normalise format of OVN agent heartbeat timestamp

    A recent change [1] to show the real heartbeat timestamp from OVN agents
    had a side effect of changing the timestamp format, which now includes a
    timezone:

        +-------------------+----------------------------------+
        | Field | Value |
        +-------------------+----------------------------------+
        | last_heartbeat_at | 2023-02-23 14:12:07.471000+00:00 |
        +-------------------+----------------------------------+

    This unexpected format change causes some clients to fail to parse the
    response to GET /v2.0/agents.

    Normalise the format of the timestamp to remove timezone information.
    Also remove the microsecond part, which was not done for OVN, but is
    absent from other network agents.

    [1] https://review.opendev.org/c/openstack/neutron/+/844179

    Closes-Bug: #2008257
    Change-Id: I75a37fb9b49a421e4524da6b56ef8362ceb6107b
    (cherry picked from commit 827fbd01c306ce292e46b0d881bc3cc804202285)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/875682
Committed: https://opendev.org/openstack/neutron/commit/19ad0f2853cf6a72b8d1c5d2e914301e52532d3e
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 19ad0f2853cf6a72b8d1c5d2e914301e52532d3e
Author: Pierre Riteau <email address hidden>
Date: Fri Feb 24 05:34:07 2023 +0100

    Normalise format of OVN agent heartbeat timestamp

    A recent change [1] to show the real heartbeat timestamp from OVN agents
    had a side effect of changing the timestamp format, which now includes a
    timezone:

        +-------------------+----------------------------------+
        | Field | Value |
        +-------------------+----------------------------------+
        | last_heartbeat_at | 2023-02-23 14:12:07.471000+00:00 |
        +-------------------+----------------------------------+

    This unexpected format change causes some clients to fail to parse the
    response to GET /v2.0/agents.

    Normalise the format of the timestamp to remove timezone information.
    Also remove the microsecond part, which was not done for OVN, but is
    absent from other network agents.

    [1] https://review.opendev.org/c/openstack/neutron/+/844179

    Closes-Bug: #2008257
    Change-Id: I75a37fb9b49a421e4524da6b56ef8362ceb6107b
    (cherry picked from commit 827fbd01c306ce292e46b0d881bc3cc804202285)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 22.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.6.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 21.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron wallaby-eom release.