[Queens -> Rocky Upgrade] python3-neutron-fwaas-dashboard installation: trying to overwrite '/etc/openstack-dashboard/neutron-fwaas-policy.json', which is also in package python-neutron-fwaas-dashboard

[Dmitrii Shcherbakov]

[Impact]
The upgrade process from Queens to Rocky fails due to a failure to override a policy file:
https://paste.ubuntu.com/p/tH4qrvP7mS/

The issue causes an upgrade action to exit prematurely which leaves the dashboard in a broken state as the keystone v3 policy json file is not rendered.

KEYSTONEV3_POLICY = ('/usr/share/openstack-dashboard/openstack_dashboard/conf/'
                     'keystonev3_policy.json')

This is reproducible without charms by simply doing the following:

lxc launch ubuntu:bionic neutrontest
Creating neutrontest
Starting neutrontest

lxc exec neutrontest bash
root@neutrontest:~# apt update
root@neutrontest:~# apt install python-neutron-fwaas-dashboard
# ... https://paste.ubuntu.com/p/RMKbNH5DbB/

root@neutrontest:~# add-apt-repository cloud-archive:rocky
root@neutrontest:~# python3-neutron-fwaas-dashboard
https://paste.ubuntu.com/p/4Fq9MvpP9p/

Unpacking python3-neutron-fwaas-dashboard (1.5.0-0ubuntu3~cloud0) ...
dpkg: error processing archive /tmp/apt-dpkg-install-R0ZsZb/111-python3-neutron-fwaas-dashboard_1.5.0-0ubuntu3~cloud0_all.deb (--unpack):
 trying to overwrite '/etc/openstack-dashboard/neutron-fwaas-policy.json', which is also in package python-neutron-fwaas-dashboard 1.3.0-0ubuntu1.1

The same issue in an output of execution of a charm action (19.04):

2019-05-08 14:38:12 DEBUG openstack-upgrade /tmp/apt-dpkg-install-3M5oD6/099-python3-neutron-fwaas-dashboard_1.5.0-0ubuntu3~cloud0_all.deb
2019-05-08 14:38:14 DEBUG openstack-upgrade E: Sub-process /usr/bin/dpkg returned an error code (1)
2019-05-08 14:38:14 INFO juju-log Couldn't acquire DPKG lock. Will retry in 10 seconds
2019-05-08 14:38:24 DEBUG openstack-upgrade Reading package lists...
2019-05-08 14:38:24 DEBUG openstack-upgrade Building dependency tree...
2019-05-08 14:38:24 DEBUG openstack-upgrade Reading state information...
2019-05-08 14:38:24 DEBUG openstack-upgrade libapache2-mod-wsgi-py3 is already the newest version (4.5.17-1).
2019-05-08 14:38:24 DEBUG openstack-upgrade python3-pymysql is already the newest version (0.8.0-1).
2019-05-08 14:38:24 DEBUG openstack-upgrade haproxy is already the newest version (1.8.8-1ubuntu0.4).
2019-05-08 14:38:24 DEBUG openstack-upgrade memcached is already the newest version (1.5.6-0ubuntu1.1).
2019-05-08 14:38:24 DEBUG openstack-upgrade python3-memcache is already the newest version (1.57-2ubuntu0.18.04.1).
2019-05-08 14:38:24 DEBUG openstack-upgrade openstack-dashboard is already the newest version (3:14.0.2-0ubuntu2~cloud0).
2019-05-08 14:38:24 DEBUG openstack-upgrade openstack-dashboard-ubuntu-theme is already the newest version (3:14.0.2-0ubuntu2~cloud0).
2019-05-08 14:38:24 DEBUG openstack-upgrade python3-designate-dashboard is already the newest version (7.0.0-0ubuntu1~cloud0).
2019-05-08 14:38:24 DEBUG openstack-upgrade python3-django-horizon is already the newest version (3:14.0.2-0ubuntu2~cloud0).
2019-05-08 14:38:24 DEBUG openstack-upgrade python3-heat-dashboard is already the newest version (1.4.0-0ubuntu1~cloud0).
2019-05-08 14:38:24 DEBUG openstack-upgrade python3-keystoneclient is already the newest version (1:3.17.0-0ubuntu1~cloud0).
2019-05-08 14:38:24 DEBUG openstack-upgrade python3-neutron-lbaas-dashboard is already the newest version (5.0.0-0ubuntu1~cloud0).
2019-05-08 14:38:24 DEBUG openstack-upgrade python3-novaclient is already the newest version (2:11.0.0-0ubuntu1~cloud0).
2019-05-08 14:38:24 DEBUG openstack-upgrade The following packages were automatically installed and are no longer required:
2019-05-08 14:38:24 DEBUG openstack-upgrade python-pika-pool python-positional python-retrying
2019-05-08 14:38:24 DEBUG openstack-upgrade Use 'apt autoremove' to remove them.
2019-05-08 14:38:24 DEBUG openstack-upgrade The following NEW packages will be installed:
2019-05-08 14:38:24 DEBUG openstack-upgrade python3-neutron-fwaas-dashboard
2019-05-08 14:38:25 DEBUG openstack-upgrade 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
2019-05-08 14:38:25 DEBUG openstack-upgrade 109 not fully installed or removed.
2019-05-08 14:38:25 DEBUG openstack-upgrade Need to get 0 B/55.6 kB of archives.
2019-05-08 14:38:25 DEBUG openstack-upgrade After this operation, 630 kB of additional disk space will be used.
(Reading database ... 81446 files and directories currently installed.)
2019-05-08 14:38:25 DEBUG openstack-upgrade Preparing to unpack .../python3-neutron-fwaas-dashboard_1.5.0-0ubuntu3~cloud0_all.deb ...
2019-05-08 14:38:25 DEBUG openstack-upgrade Unpacking python3-neutron-fwaas-dashboard (1.5.0-0ubuntu3~cloud0) ...
2019-05-08 14:38:25 DEBUG openstack-upgrade dpkg: error processing archive /var/cache/apt/archives/python3-neutron-fwaas-dashboard_1.5.0-0ubuntu3~cloud0_all.deb (--unpack):
2019-05-08 14:38:25 DEBUG openstack-upgrade trying to overwrite '/etc/openstack-dashboard/neutron-fwaas-policy.json', which is also in package python-neutron-fwaas-dashboard 1.3.0-0ubuntu1.1
2019-05-08 14:38:25 DEBUG openstack-upgrade Errors were encountered while processing:
2019-05-08 14:38:25 DEBUG openstack-upgrade /var/cache/apt/archives/python3-neutron-fwaas-dashboard_1.5.0-0ubuntu3~cloud0_all.deb
2019-05-08 14:38:27 DEBUG openstack-upgrade E: Sub-process /usr/bin/dpkg returned an error code (1)
2019-05-08 14:38:27 INFO juju-log Couldn't acquire DPKG lock. Will retry in 10 seconds

[Test Case]
# apt update
# apt install python-neutron-fwaas-dashboard
# add-apt-repository cloud-archive:rocky
# apt dist-upgrade
# apt install python3-neutron-fwaas-dashboard

[Regression Potential]
Regression potential is low. The fix is just adding a breaks/replaces to python3-neutron-fwaas-dashboard to ensure python-neutron-fwaas-dashboard is removed when python3-neutron-fwaas-dashboard is installed.

[Dmitrii Shcherbakov (dmitriis)]

The charm bug is about not reporting a failed upgrade in its status.

openstack-dashboard/0 active idle 3/lxd/6 10.232.46.158 80/tcp,443/tcp Unit is ready
  hacluster-horizon/1 active idle 10.232.46.158 Unit is ready and clustered
  octavia-dashboard/1 active idle 10.232.46.158 Unit is ready
openstack-dashboard/1* active idle 4/lxd/3 10.232.46.146 80/tcp,443/tcp Unit is ready
  hacluster-horizon/0* active idle 10.232.46.146 Unit is ready and clustered
  octavia-dashboard/0* active idle 10.232.46.146 Unit is ready

[Corey Bryant (corey.bryant)]

Hi Dmitrii,

Thanks for reporting this.

I was able to recreate this on bionic with:

# apt update
# apt install python-neutron-fwaas-dashboard
# add-apt-repository cloud-archive:rocky
# apt dist-upgrade
# python3-neutron-fwaas-dashboard

Triaging this as high and I'll work on a fix.

Corey

[Corey Bryant (corey.bryant)]

That last step should've been:

# apt install python3-neutron-fwaas-dashboard

[Corey Bryant (corey.bryant)]

Fixed versions of neutron-fwaas-dashboard have been uploaded to eoan as well as disco/cosmic unapproved queues [1] where they are awaiting SRU team review.

[1]
https://launchpad.net/ubuntu/disco/+queue?queue_state=1&queue_text=neutron-fwaas-dashboard
https://launchpad.net/ubuntu/cosmic/+queue?queue_state=1&queue_text=neutron-fwaas-dashboard

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron-fwaas-dashboard - 2.0.1-0ubuntu2

---------------
neutron-fwaas-dashboard (2.0.1-0ubuntu2) eoan; urgency=medium

  * d/control: Add Breaks/Replaces to python3-neutron-fwaas-dashboard to
    ensure python-neutron-fwaas-dashboard is removed on upgrade (LP: #1828293).
  * d/control: (Build-)Depends on latest python3-django-horizon and
    openstack-dashboard to ensure unit tests are successful (LP: #1806053).

 -- Corey Bryant <email address hidden> Thu, 09 May 2019 22:05:46 -0400

[David Ames (thedac)]

CHARM TRIAGE:

Per comment #1 "The charm bug is about not reporting a failed upgrade in its status."

[Łukasz Zemczak (sil2100)]

Hello Dmitrii, or anyone else affected,

Accepted neutron-fwaas-dashboard into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron-fwaas-dashboard/2.0.1-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello Dmitrii, or anyone else affected,

Accepted neutron-fwaas-dashboard into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Dmitrii, or anyone else affected,

Accepted neutron-fwaas-dashboard into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron-fwaas-dashboard/1.5.0-0ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello Dmitrii, or anyone else affected,

Accepted neutron-fwaas-dashboard into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hi Dmitrii, if you get a chance, please can you test that this is fixed in -proposed and tag this bug accordingly (ie. verification-done-*, etc)? Thanks!

[Dmitrii Shcherbakov (dmitriis)]

I can confirm that this no longer occurs with proposed.

apt policy python3-neutron-fwaas-dashboard
python3-neutron-fwaas-dashboard:
  Installed: 2.0.1-0ubuntu1.1~cloud0
  Candidate: 2.0.1-0ubuntu1.1~cloud0

I did run into a different problem though for which I filed a separate bug:

https://bugs.launchpad.net/charm-openstack-dashboard/+bug/1832075

[Dmitrii Shcherbakov (dmitriis)]

Verified with a Queens -> Rocky upgrade with a workaround for lp:1832075 (pre-installing python3-pymysql).

apt policy python3-neutron-fwaas-dashboard
python3-neutron-fwaas-dashboard:
  Installed: 1.5.0-0ubuntu3.1~cloud0
  Candidate: 1.5.0-0ubuntu3.1~cloud0
  Version table:
 *** 1.5.0-0ubuntu3.1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/rocky/main amd64 Packages
        100 /var/lib/dpkg/status
     1.5.0-0ubuntu3~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/rocky/main amd64 Packages

[Corey Bryant (corey.bryant)]

python-neutron-fwaas-dashboard binary package is not available after Bionic so testing for Cosmic and Disco will just involve installation of python3-neutron-fwaas-dashboard to ensure no regression.

[Corey Bryant (corey.bryant)]

Successfully tested for cosmic-proposed and disco-proposed:

root@c1:~# apt policy python3-neutron-fwaas-dashboard
python3-neutron-fwaas-dashboard:
  Installed: (none)
  Candidate: 1.5.0-0ubuntu3.1
  Version table:
     1.5.0-0ubuntu3.1 500
        500 http://archive.ubuntu.com/ubuntu cosmic-proposed/universe amd64 Packages
     1.5.0-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu cosmic/universe amd64 Packages
root@c1:~# sudo apt install --yes python3-neutron-fwaas-dashboard
...
root@c1:~# apt policy python3-neutron-fwaas-dashboard
python3-neutron-fwaas-dashboard:
  Installed: 1.5.0-0ubuntu3.1
  Candidate: 1.5.0-0ubuntu3.1
  Version table:
 *** 1.5.0-0ubuntu3.1 500
        500 http://archive.ubuntu.com/ubuntu cosmic-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     1.5.0-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu cosmic/universe amd64 Packages

python3-neutron-fwaas-dashboard:
  Installed: (none)
  Candidate: 2.0.1-0ubuntu1.1
  Version table:
     2.0.1-0ubuntu1.1 500
        500 http://archive.ubuntu.com/ubuntu disco-proposed/universe amd64 Packages
     2.0.1-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu disco/universe amd64 Packages
root@d2:~# sudo apt install --yes python3-neutron-fwaas-dashboard
...
root@d2:~# apt policy python3-neutron-fwaas-dashboard:
python3-neutron-fwaas-dashboard:
  Installed: 2.0.1-0ubuntu1.1
  Candidate: 2.0.1-0ubuntu1.1
  Version table:
 *** 2.0.1-0ubuntu1.1 500
        500 http://archive.ubuntu.com/ubuntu disco-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     2.0.1-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu disco/universe amd64 Packages

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron-fwaas-dashboard - 2.0.1-0ubuntu1.1

---------------
neutron-fwaas-dashboard (2.0.1-0ubuntu1.1) disco; urgency=medium

  * d/gbp.conf: Create stable/stein branch.
  * d/control: Add Breaks/Replaces to python3-neutron-fwaas-dashboard to
    ensure python-neutron-fwaas-dashboard is removed on upgrade (LP: #1828293).
  * d/control: (Build-)Depends on latest python3-django-horizon and
    openstack-dashboard to ensure unit tests are successful (LP: #1806053).

 -- Corey Bryant <email address hidden> Tue, 07 May 2019 14:15:42 -0400

[Robie Basak (racb)]

The verification of the Stable Release Update for neutron-fwaas-dashboard has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron-fwaas-dashboard - 1.5.0-0ubuntu3.1

---------------
neutron-fwaas-dashboard (1.5.0-0ubuntu3.1) cosmic; urgency=medium

  * d/gbp.conf: Create stable/rocky branch.
  * d/control: Add Breaks/Replaces to python3-neutron-fwaas-dashboard to
    ensure python-neutron-fwaas-dashboard is removed on upgrade (LP: #1828293).
  * d/control: (Build-)Depends on latest python3-django-horizon and
    openstack-dashboard to ensure unit tests are successful (LP: #1806053).

 -- Corey Bryant <email address hidden> Wed, 31 Oct 2018 14:10:53 -0400

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron-fwaas-dashboard has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron-fwaas-dashboard - 2.0.1-0ubuntu1.1~cloud0
---------------

 neutron-fwaas-dashboard (2.0.1-0ubuntu1.1~cloud0) bionic-stein; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron-fwaas-dashboard (2.0.1-0ubuntu1.1) disco; urgency=medium
 .
   * d/gbp.conf: Create stable/stein branch.
   * d/control: Add Breaks/Replaces to python3-neutron-fwaas-dashboard to
     ensure python-neutron-fwaas-dashboard is removed on upgrade (LP: #1828293).
   * d/control: (Build-)Depends on latest python3-django-horizon and
     openstack-dashboard to ensure unit tests are successful (LP: #1806053).

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron-fwaas-dashboard has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron-fwaas-dashboard - 1.5.0-0ubuntu3.1~cloud0
---------------

 neutron-fwaas-dashboard (1.5.0-0ubuntu3.1~cloud0) bionic-rocky; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron-fwaas-dashboard (1.5.0-0ubuntu3.1) cosmic; urgency=medium
 .
   * d/gbp.conf: Create stable/rocky branch.
   * d/control: Add Breaks/Replaces to python3-neutron-fwaas-dashboard to
     ensure python-neutron-fwaas-dashboard is removed on upgrade (LP: #1828293).
   * d/control: (Build-)Depends on latest python3-django-horizon and
     openstack-dashboard to ensure unit tests are successful (LP: #1806053).

[Dmitrii Shcherbakov (dmitriis)]

Looks like Rocky UCA hasn't been updated with 1.5.0-0ubuntu3.1 yet:

http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/bionic-updates/rocky/main/binary-amd64/Packages

Package: python3-neutron-fwaas-dashboard
Source: neutron-fwaas-dashboard
Priority: optional
Section: python
Installed-Size: 615
Maintainer: Ubuntu Developers <email address hidden>
Architecture: all
Version: 1.5.0-0ubuntu3~cloud0

apt policy python3-neutron-fwaas-dashboard
python3-neutron-fwaas-dashboard:
  Installed: (none)
  Candidate: 1.5.0-0ubuntu3~cloud0
  Version table:
     1.5.0-0ubuntu3~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/rocky/main amd64 Packages

[Corey Bryant (corey.bryant)]

@Dmitrii, I looked at a couple of OpenStack charms and they didn't appear to handle upgrade failures in the hooks/*utils.py or actions/openstack_upgrade.py. Would you be opposed to moving that part of this bug to a new bug where we can track the issue across all affected charms?

[Dmitrii Shcherbakov (dmitriis)]

Corey,

Created a separate bug, feel free to add other affected charms https://bugs.launchpad.net/charm-openstack-dashboard/+bug/1837457

[Alex Kavanagh (ajkavanagh)]

Marking the openstack-dashboard bug as invalid, as it was a packaging bug. There is a separate bug (lp:1837457) about handling upgrade failures in charms as they were not reported.