[Focal] Policy files are missing

[Albert Damen]

[Impact]
[Test Case]
python3-django-horizon:
  Installed: 3:18.2.1~git2020032709.2c4470272-0ubuntu1

After a fresh install of openstack dashboard on focal, apache2 error.log contains hundreds of error message about missing policy files:

[Tue Apr 14 09:12:34.558183 2020] [wsgi:error] [pid 3062:tid 140253993006848] [remote 10.64.255.1:50364] WARNING openstack_auth.policy No policy rules for service 'identity' in /usr/lib/python3/dist-packages/openstack_dashboard/conf/keystone_policy.json
[Tue Apr 14 09:12:34.559486 2020] [wsgi:error] [pid 3062:tid 140253993006848] [remote 10.64.255.1:50364] WARNING openstack_auth.policy No policy rules for service 'compute' in /usr/lib/python3/dist-packages/openstack_dashboard/conf/nova_policy.json and files under ['/usr/lib/python3/dist-packages/openstack_dashboard/conf/nova_policy.d']
[Tue Apr 14 09:12:34.560622 2020] [wsgi:error] [pid 3062:tid 140253993006848] [remote 10.64.255.1:50364] WARNING openstack_auth.policy No policy rules for service 'volume' in /usr/lib/python3/dist-packages/openstack_dashboard/conf/cinder_policy.json and files under ['/usr/lib/python3/dist-packages/openstack_dashboard/conf/cinder_policy.d']
[Tue Apr 14 09:12:34.561703 2020] [wsgi:error] [pid 3062:tid 140253993006848] [remote 10.64.255.1:50364] WARNING openstack_auth.policy No policy rules for service 'image' in /usr/lib/python3/dist-packages/openstack_dashboard/conf/glance_policy.json
[Tue Apr 14 09:12:34.562703 2020] [wsgi:error] [pid 3062:tid 140253993006848] [remote 10.64.255.1:50364] WARNING openstack_auth.policy No policy rules for service 'network' in /usr/lib/python3/dist-packages/openstack_dashboard/conf/neutron_policy.json

The policy files are indeed missing from the package:

dpkg -L python3-django-horizon | grep json$
/usr/lib/python3/dist-packages/horizon/xstatic/pkg/angular/data/errors.json
/usr/lib/python3/dist-packages/horizon/xstatic/pkg/angular/data/version.json
/usr/lib/python3/dist-packages/horizon-18.2.1.dev1.egg-info/pbr.json

Logging in with a normal user account (without admin role) still shows the admin panel and buttons a normal user cannot use, like identity/users "create user". Trying to use these either doesn't work or throws errors like "Unable to retrieve xxx".

Copying the policy files from the source package solves the problem.

[Regression Potential]
Very low, this patch is just adding necessary policy files back into the package that have already existed in prior releases.

[Albert Damen (albrt)]

This issue seems to be caused by the regenerated SOURCES.txt file from:

horizon (3:18.0.1~git2020021409.bb959361b-0ubuntu3) focal; urgency=medium

  * d/rules: Force regeneration of SOURCES.txt to ensure that generated
    django{js}.mo files are included in binary packages, resolving issues
    with translations (LP: #1862343).

That change includes .mo files in the package, but at the seem time removed a number of files from the package, see the attached file.

[Albert Damen (albrt)]

This is still an issue with the package from focal-proposed: 3:18.3.2-0ubuntu0.20.04.1

[Albert Damen (albrt)]

Trying to build the focal proposed package, it seems adding the mo files to SOURCES.txt with sed instead of regenerating SOURCES.txt does the job.

I changed debian/rules:
#rm -f horizon.egg-info/SOURCES.txt
sed -i 's/\(.*\)po$$/\1po\n\1mo/' horizon.egg-info/SOURCES.txt

Building the package with the original sources.txt, regenerating sources.txt and changed sources.txt gives:

Counting non .mo files in python3-django-horizon_18.3.2-0ubuntu0.20.04.1_all.deb
Original SOURCES.txt: 4335
Regenerated SOURCES.txt: 4301
Changed SOURCES.txt: 4335

Counting .mo files
Original SOURCES.txt: 0
Regenerated SOURCES.txt: 96
Changed SOURCES.txt: 96

So changing SOURCES.txt adds the same number of .mo files without losing any non .mo file.

Also it doesn't loose any .json files:
Original SOURCES.txt: 21
Regenerated SOURCES.txt: 3
Changed SOURCES.txt: 21

The resulting package works fine and solves the problems for non-admin users.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Albert Damen (albrt)]

Also happens with bionic/ussuri

apt policy python3-django-horizon
python3-django-horizon:
  Installed: 3:18.3.2-0ubuntu0.20.04.1~cloud0
  Candidate: 3:18.3.2-0ubuntu0.20.04.1~cloud0
  Version table:
 *** 3:18.3.2-0ubuntu0.20.04.1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/ussuri/main amd64 Packages
        100 /var/lib/dpkg/status

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/734075

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/734075
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=f7dc499de44a575fa2966443784af3d14efcc6ae
Submitter: Zuul
Branch: master

commit f7dc499de44a575fa2966443784af3d14efcc6ae
Author: Chris MacNaughton <email address hidden>
Date: Mon Jun 8 15:06:17 2020 +0200

    Update MANIFEST for dashboard config

    The various policy files in openstack-dashboard/conf are not
    included in the package by default. Update MANIFEST.in to
    recursively include that directory.

    Change-Id: I889a639b5d8d696df997c031036ffb8f88aa66b9
    Closes-Bug: #1872671

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/735310

[Launchpad Janitor (janitor)]

This bug was fixed in the package horizon - 3:18.3.2-0ubuntu2

---------------
horizon (3:18.3.2-0ubuntu2) groovy; urgency=medium

  * d/p/fix-skipped-config-files.patch: Ensure that config files
    are included in the package (LP: #1872671).

 -- Chris MacNaughton <email address hidden> Fri, 12 Jun 2020 11:20:25 +0100

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/735310
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=bed596616959e649e6d7215dc136c1f7616cc532
Submitter: Zuul
Branch: stable/ussuri

commit bed596616959e649e6d7215dc136c1f7616cc532
Author: Chris MacNaughton <email address hidden>
Date: Mon Jun 8 15:06:17 2020 +0200

    Update MANIFEST for dashboard config

    The various policy files in openstack-dashboard/conf are not
    included in the package by default. Update MANIFEST.in to
    recursively include that directory.

    Change-Id: I889a639b5d8d696df997c031036ffb8f88aa66b9
    Closes-Bug: #1872671
    (cherry picked from commit f7dc499de44a575fa2966443784af3d14efcc6ae)

[Corey Bryant (corey.bryant)]

This bug was fixed in the package horizon - 3:18.4.2~git2020070209.392bc2482-0ubuntu1~cloud0
---------------

 horizon (3:18.4.2~git2020070209.392bc2482-0ubuntu1~cloud0) focal-victoria; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 horizon (3:18.4.2~git2020070209.392bc2482-0ubuntu1) groovy; urgency=medium
 .
   * New upstream snapshot for OpenStack Victoria.
   * d/control: Align (Build-)Depends with upstream.
   * d/p/fix-skipped-config-files.patch: Dropped. Fixed upstream.
   * d/control: Update Standards-Version to 4.5.0.
 .
 horizon (3:18.3.2-0ubuntu2) groovy; urgency=medium
 .
   * d/p/fix-skipped-config-files.patch: Ensure that config files
     are included in the package (LP: #1872671).
 .
 horizon (3:18.3.2-0ubuntu1) groovy; urgency=medium
 .
   * New upstream release for OpenStack Ussuri (LP: #1877642).

[Corey Bryant (corey.bryant)]

This Chris. Merged and uploaded to focal unapproved queue.

[Robie Basak (racb)]

Hello Albert, or anyone else affected,

Accepted horizon into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/horizon/3:18.3.2-0ubuntu0.20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Albert Damen (albrt)]

I created a new horizon host on focal and installed openstack-dashboard.

openstack-dashboard:
  Installed: 3:18.3.2-0ubuntu0.20.04.1

apache error log showed many errors about missing policy files
admin panel was visible while logged in as non-admin user

I enabled focal-proposed and upgraded openstack-dashboard

openstack-dashboard:
  Installed: 3:18.3.2-0ubuntu0.20.04.2

no error messages about missing policy files
admin panel was not visible for non-admin user
started and deleted new instance as non-admin user
switched to admin user and verified admin operations still worked

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for horizon has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package horizon - 3:18.3.2-0ubuntu0.20.04.2

---------------
horizon (3:18.3.2-0ubuntu0.20.04.2) focal; urgency=medium

  * d/p/fix-skipped-config-files.patch: Ensure that policy files are
    included in the package (LP: #1872671).

 -- Chris MacNaughton <email address hidden> Wed, 01 Jul 2020 14:54:51 +0000

[Corey Bryant (corey.bryant)]

Hello Albert, or anyone else affected,

Accepted horizon into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Albert Damen (albrt)]

I created a new horizon host on bionic with bionic/ussuri and installed openstack-dashboard.

openstack-dashboard:
  Installed: 3:18.3.2-0ubuntu0.20.04.1~cloud0

apache error log showed many errors about missing policy files
admin panel was visible while logged in as non-admin user

I enabled bionic/ussuri-proposed and upgraded openstack-dashboard

openstack-dashboard:
  Installed: 3:18.3.2-0ubuntu0.20.04.2~cloud0

no error messages about missing policy files
admin panel was not visible for non-admin user
started and deleted new instance as non-admin user
switched to admin user and verified admin operations still worked

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for horizon has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package horizon - 3:18.3.2-0ubuntu0.20.04.2~cloud0
---------------

 horizon (3:18.3.2-0ubuntu0.20.04.2~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 horizon (3:18.3.2-0ubuntu0.20.04.2) focal; urgency=medium
 .
   * d/p/fix-skipped-config-files.patch: Ensure that policy files are
     included in the package (LP: #1872671).

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 18.3.3 release.