Activity log for bug #1907081

Date Who What changed Old value New value Message
2020-12-07 11:39:07 Frode Nordahl bug added bug
2020-12-07 11:39:18 Frode Nordahl ovn (Ubuntu): status New Triaged
2020-12-07 11:39:23 Frode Nordahl ovn (Ubuntu): importance Undecided Medium
2020-12-07 11:39:31 Frode Nordahl bug task added charm-ovn-central
2020-12-07 11:39:51 Frode Nordahl charm-ovn-central: status New Triaged
2020-12-07 11:39:55 Frode Nordahl charm-ovn-central: importance Undecided Medium
2020-12-07 11:53:02 Frode Nordahl charm-ovn-central: status Triaged In Progress
2020-12-07 11:53:05 Frode Nordahl charm-ovn-central: assignee Frode Nordahl (fnordahl)
2020-12-07 11:55:05 Frode Nordahl summary OVN database is not upgraded on package upgrade Clustered OVN database is not upgraded on package upgrade
2020-12-07 18:22:51 Frode Nordahl ovn (Ubuntu): assignee Frode Nordahl (fnordahl)
2020-12-11 11:26:35 Launchpad Janitor merge proposal linked https://code.launchpad.net/~fnordahl/ubuntu/+source/ovn/+git/ovn/+merge/394584
2020-12-11 11:27:33 Frode Nordahl ovn (Ubuntu): status Triaged In Progress
2020-12-11 11:53:03 Launchpad Janitor merge proposal linked https://code.launchpad.net/~fnordahl/ubuntu/+source/ovn/+git/ovn/+merge/395221
2021-01-12 11:46:58 James Page bug added subscriber Ubuntu Stable Release Updates Team
2021-01-12 11:53:19 James Page description In the systemd service we make use of the `ovn-ctl` script `run_nb_ovsdb` and `run_sb_ovsdb` sub-commands introduced in [0]. These sub-commands fit nicely with systemd's expectations of modern daemons to no longer detachand run in the background. However, the change in [0] has the side effect of disabling automatic upgrading of clustered databases. Previously this would have been done on every startup [1]. A recent commit to master [2] addresses this and uses the combination of presence of `--db-*-cluster-local-addr` and non-presence of the `--db-*-cluster-remote-addr` to determine if the upgrade should be run. We should backport [2] to our supported OVN packages to prepare for supporting upgrades that require database schema changes. We may also need to change the behavior of the ovn-central charm to not set the `--db-*-cluster-remote-addr` argument on the leader unit. 0: https://github.com/ovn-org/ovn/commit/6444059b5f9444ce06634794d275257f945a6ce5 1: https://github.com/ovn-org/ovn/blob/5c2d311b8b7b4d5c3a619de72be6a433aa4c44db/utilities/ovn-ctl#L312-L314 2: https://github.com/ovn-org/ovn/commit/67e2f386cc838d0b0f9b4b5da7fe611e1113b70c [Impact] [Test Case] [Regression Potential] [Original Bug Report] In the systemd service we make use of the `ovn-ctl` script `run_nb_ovsdb` and `run_sb_ovsdb` sub-commands introduced in [0]. These sub-commands fit nicely with systemd's expectations of modern daemons to no longer detachand run in the background. However, the change in [0] has the side effect of disabling automatic upgrading of clustered databases. Previously this would have been done on every startup [1]. A recent commit to master [2] addresses this and uses the combination of presence of `--db-*-cluster-local-addr` and non-presence of the `--db-*-cluster-remote-addr` to determine if the upgrade should be run. We should backport [2] to our supported OVN packages to prepare for supporting upgrades that require database schema changes. We may also need to change the behavior of the ovn-central charm to not set the `--db-*-cluster-remote-addr` argument on the leader unit. 0: https://github.com/ovn-org/ovn/commit/6444059b5f9444ce06634794d275257f945a6ce5 1: https://github.com/ovn-org/ovn/blob/5c2d311b8b7b4d5c3a619de72be6a433aa4c44db/utilities/ovn-ctl#L312-L314 2: https://github.com/ovn-org/ovn/commit/67e2f386cc838d0b0f9b4b5da7fe611e1113b70c
2021-01-12 13:39:35 Frode Nordahl description [Impact] [Test Case] [Regression Potential] [Original Bug Report] In the systemd service we make use of the `ovn-ctl` script `run_nb_ovsdb` and `run_sb_ovsdb` sub-commands introduced in [0]. These sub-commands fit nicely with systemd's expectations of modern daemons to no longer detachand run in the background. However, the change in [0] has the side effect of disabling automatic upgrading of clustered databases. Previously this would have been done on every startup [1]. A recent commit to master [2] addresses this and uses the combination of presence of `--db-*-cluster-local-addr` and non-presence of the `--db-*-cluster-remote-addr` to determine if the upgrade should be run. We should backport [2] to our supported OVN packages to prepare for supporting upgrades that require database schema changes. We may also need to change the behavior of the ovn-central charm to not set the `--db-*-cluster-remote-addr` argument on the leader unit. 0: https://github.com/ovn-org/ovn/commit/6444059b5f9444ce06634794d275257f945a6ce5 1: https://github.com/ovn-org/ovn/blob/5c2d311b8b7b4d5c3a619de72be6a433aa4c44db/utilities/ovn-ctl#L312-L314 2: https://github.com/ovn-org/ovn/commit/67e2f386cc838d0b0f9b4b5da7fe611e1113b70c [Impact] On upgrade of the OVN packages it may be necessary to perform a upgrade to the Northbound and Southbound databases. Failure to do so may lead to loss of connectivity between participating nodes as the software components will attempt to make use of columns that are not available in the database. The upgrade process has been performed automatically by the upstream init script by default since inception, both for a local and clustered setup. But as discussed below recent changes has inadvertently omitted this behavior. [Test Case] Non-clustered scenario: Install the ovn-central components in a container using the in-release focal package and start the database and ovn-northd services. Upgrade the container to the OVN packages from in-release Groovy and observe the ovn-northd service complaining about missing columns in the database. Repeat the test with the proposed packages and observe that the databases being upgraded on restart of services after package upgrade. Clustered scenario: Install the ovn-central charm across three containers and necessary dependencies. Perform package upgrade as outlined above and compare how in-relase and proposed packages behave. [Regression Potential] As we are restoring the intended behavior the regression potential is minimal. [Original Bug Report] In the systemd service we make use of the `ovn-ctl` script `run_nb_ovsdb` and `run_sb_ovsdb` sub-commands introduced in [0]. These sub-commands fit nicely with systemd's expectations of modern daemons to no longer detachand run in the background. However, the change in [0] has the side effect of disabling automatic upgrading of clustered databases. Previously this would have been done on every startup [1]. A recent commit to master [2] addresses this and uses the combination of presence of `--db-*-cluster-local-addr` and non-presence of the `--db-*-cluster-remote-addr` to determine if the upgrade should be run. We should backport [2] to our supported OVN packages to prepare for supporting upgrades that require database schema changes. We may also need to change the behavior of the ovn-central charm to not set the `--db-*-cluster-remote-addr` argument on the leader unit. 0: https://github.com/ovn-org/ovn/commit/6444059b5f9444ce06634794d275257f945a6ce5 1: https://github.com/ovn-org/ovn/blob/5c2d311b8b7b4d5c3a619de72be6a433aa4c44db/utilities/ovn-ctl#L312-L314 2: https://github.com/ovn-org/ovn/commit/67e2f386cc838d0b0f9b4b5da7fe611e1113b70c
2021-01-12 13:46:13 Frode Nordahl description [Impact] On upgrade of the OVN packages it may be necessary to perform a upgrade to the Northbound and Southbound databases. Failure to do so may lead to loss of connectivity between participating nodes as the software components will attempt to make use of columns that are not available in the database. The upgrade process has been performed automatically by the upstream init script by default since inception, both for a local and clustered setup. But as discussed below recent changes has inadvertently omitted this behavior. [Test Case] Non-clustered scenario: Install the ovn-central components in a container using the in-release focal package and start the database and ovn-northd services. Upgrade the container to the OVN packages from in-release Groovy and observe the ovn-northd service complaining about missing columns in the database. Repeat the test with the proposed packages and observe that the databases being upgraded on restart of services after package upgrade. Clustered scenario: Install the ovn-central charm across three containers and necessary dependencies. Perform package upgrade as outlined above and compare how in-relase and proposed packages behave. [Regression Potential] As we are restoring the intended behavior the regression potential is minimal. [Original Bug Report] In the systemd service we make use of the `ovn-ctl` script `run_nb_ovsdb` and `run_sb_ovsdb` sub-commands introduced in [0]. These sub-commands fit nicely with systemd's expectations of modern daemons to no longer detachand run in the background. However, the change in [0] has the side effect of disabling automatic upgrading of clustered databases. Previously this would have been done on every startup [1]. A recent commit to master [2] addresses this and uses the combination of presence of `--db-*-cluster-local-addr` and non-presence of the `--db-*-cluster-remote-addr` to determine if the upgrade should be run. We should backport [2] to our supported OVN packages to prepare for supporting upgrades that require database schema changes. We may also need to change the behavior of the ovn-central charm to not set the `--db-*-cluster-remote-addr` argument on the leader unit. 0: https://github.com/ovn-org/ovn/commit/6444059b5f9444ce06634794d275257f945a6ce5 1: https://github.com/ovn-org/ovn/blob/5c2d311b8b7b4d5c3a619de72be6a433aa4c44db/utilities/ovn-ctl#L312-L314 2: https://github.com/ovn-org/ovn/commit/67e2f386cc838d0b0f9b4b5da7fe611e1113b70c [Impact] On upgrade of the OVN packages it may be necessary to perform a upgrade to the Northbound and Southbound databases. Failure to do so may lead to loss of connectivity between participating nodes as the software components will attempt to make use of columns that are not available in the database. The upgrade process has been performed automatically by the upstream init script by default since inception, both for a local and clustered setup. But as discussed below recent changes has inadvertently omitted this behavior for clustered databases. [Test Case] Non-clustered scenario as reference test: Install the ovn-central package in a container using the in-release focal package and start the database and ovn-northd services. Upgrade the container to the OVN packages from in-release Groovy and observe the package performing the database upgrade and subsequently ovn-northd service not complaining about missing columns in the database. Clustered scenario: Install the ovn-central charm across three containers and necessary dependencies. Perform package upgrade as outlined above and compare how in-relase and proposed packages behave. [Regression Potential] As we are restoring the intended behavior the regression potential is minimal. [Original Bug Report] In the systemd service we make use of the `ovn-ctl` script `run_nb_ovsdb` and `run_sb_ovsdb` sub-commands introduced in [0]. These sub-commands fit nicely with systemd's expectations of modern daemons to no longer detachand run in the background. However, the change in [0] has the side effect of disabling automatic upgrading of clustered databases. Previously this would have been done on every startup [1]. A recent commit to master [2] addresses this and uses the combination of presence of `--db-*-cluster-local-addr` and non-presence of the `--db-*-cluster-remote-addr` to determine if the upgrade should be run. We should backport [2] to our supported OVN packages to prepare for supporting upgrades that require database schema changes. We may also need to change the behavior of the ovn-central charm to not set the `--db-*-cluster-remote-addr` argument on the leader unit. 0: https://github.com/ovn-org/ovn/commit/6444059b5f9444ce06634794d275257f945a6ce5 1: https://github.com/ovn-org/ovn/blob/5c2d311b8b7b4d5c3a619de72be6a433aa4c44db/utilities/ovn-ctl#L312-L314 2: https://github.com/ovn-org/ovn/commit/67e2f386cc838d0b0f9b4b5da7fe611e1113b70c
2021-01-12 14:22:01 Frode Nordahl charm-ovn-central: status In Progress Fix Committed
2021-01-12 14:22:08 Frode Nordahl charm-ovn-central: milestone 21.01
2021-01-12 14:22:41 Frode Nordahl nominated for series Ubuntu Focal
2021-01-12 14:22:41 Frode Nordahl bug task added ovn (Ubuntu Focal)
2021-01-12 14:22:41 Frode Nordahl nominated for series Ubuntu Groovy
2021-01-12 14:22:41 Frode Nordahl bug task added ovn (Ubuntu Groovy)
2021-01-12 14:22:41 Frode Nordahl nominated for series Ubuntu Hirsute
2021-01-12 14:22:41 Frode Nordahl bug task added ovn (Ubuntu Hirsute)
2021-01-12 14:22:56 Frode Nordahl ovn (Ubuntu Hirsute): status In Progress Fix Released
2021-01-12 14:23:01 Frode Nordahl ovn (Ubuntu Hirsute): assignee Frode Nordahl (fnordahl)
2021-01-12 14:23:06 Frode Nordahl ovn (Ubuntu Groovy): status New In Progress
2021-01-12 14:23:12 Frode Nordahl ovn (Ubuntu Focal): status New In Progress
2021-01-12 14:23:17 Frode Nordahl ovn (Ubuntu Groovy): importance Undecided Medium
2021-01-12 14:23:20 Frode Nordahl ovn (Ubuntu Focal): importance Undecided Medium
2021-01-12 14:23:23 Frode Nordahl ovn (Ubuntu Groovy): assignee Frode Nordahl (fnordahl)
2021-01-12 14:23:27 Frode Nordahl ovn (Ubuntu Focal): assignee Frode Nordahl (fnordahl)
2021-01-19 19:12:56 Brian Murray ovn (Ubuntu Groovy): status In Progress Fix Committed
2021-01-19 19:12:59 Brian Murray bug added subscriber SRU Verification
2021-01-19 19:13:03 Brian Murray tags verification-needed verification-needed-groovy
2021-01-19 19:22:53 Brian Murray ovn (Ubuntu Focal): status In Progress Fix Committed
2021-01-19 19:23:00 Brian Murray tags verification-needed verification-needed-groovy verification-needed verification-needed-focal verification-needed-groovy
2021-02-10 22:50:16 David Ames charm-ovn-central: status Fix Committed Fix Released
2021-02-15 17:03:44 Frode Nordahl tags verification-needed verification-needed-focal verification-needed-groovy verification-done-focal verification-needed verification-needed-groovy
2021-02-15 17:12:30 Frode Nordahl tags verification-done-focal verification-needed verification-needed-groovy verification-done verification-done-focal verification-done-groovy
2021-02-18 09:19:40 Launchpad Janitor ovn (Ubuntu Groovy): status Fix Committed Fix Released
2021-02-18 09:19:50 Ɓukasz Zemczak removed subscriber Ubuntu Stable Release Updates Team
2021-02-18 09:26:48 Launchpad Janitor ovn (Ubuntu Focal): status Fix Committed Fix Released