Can't launch instances / Forbidden (CSRF token missing or incorrect.): /api/policy/

[TWENTY |20]

Cloud: bionic-queens

I use the horizon dashboard over https with ssl-cert and without any problemes before.

After I made a charm-upgrade to revision 294 I can't launch instances in the horizon dashboard anymore.

I found some errors in the apache logfile:

DEBUG:urllib3.connectionpool:Resetting dropped connection: nova-cloud-controller-internalapi.....
Forbidden (CSRF token missing or incorrect.): /api/policy/

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): neutron-api-internalapi....
Forbidden (CSRF token missing or incorrect.): /api/policy/

I tested this with revision 295 and there is still this problem.

I can launch instances via the CLI without problems, that's why I think it's just a horizon problem.

If I downgrade back to revision 293, the issue is gone.
(juju upgrade-charm openstack-dashboard --revision 293)

[Alex Kavanagh (ajkavanagh)]

Hi

I'm wondering if this is related to this fix:

commit 101098a1c2c1d7e30f5d4406e599074dee5d3ce7
Author: Sahid Orentino Ferdjaoui <email address hidden>
Date: Tue Apr 2 12:14:23 2019 +0200

    apache2: add secure flag header when enforce_ssl

    The Secure attribute tells the browser to only send the cookie if the
    request is being sent over a secure channel such as HTTPS. This will
    help protect the cookie from being passed over unencrypted requests.

    Change-Id: I1ded951d79ad9fa832d1e88f656a1e064b1ef007
    Closes-bug: #1822751
    Signed-off-by: Sahid Orentino Ferdjaoui <email address hidden>

diff --git a/templates/default-ssl b/templates/default-ssl
index c6caf22..1b42cf5 100644
--- a/templates/default-ssl
+++ b/templates/default-ssl
@@ -44,6 +44,7 @@ NameVirtualHost *:{{ 443 }}
     SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
 {% if enforce_ssl %}
     Header set Strict-Transport-Security "max-age={{ hsts_max_age_seconds }}"
+ Header edit Set-Cookie ^(.*)$ $1;HttpOnly;Secure
 {% endif %}
     Header set X-XSS-Protection "1; mode=block"
     Header set X-Content-Type-Options "nosniff"

(related bug: https://bugs.launchpad.net/charm-openstack-dashboard/+bug/1822751)

I ask due to this issue (https://github.com/opensourcepos/opensourcepos/issues/1492) which talks about a similar thing happening. As openstack-dashboard uses Angular it could be related to the CSRF token not being readable by the JS in the app?

[David Coronel (davecore)]

I experienced the same issue. This is a bionic queens cloud with the openstack-dashboard charm rev 295.

I did have the enforce-ssl charm option set to True in the openstack-dashboard charm.

I disabled it and now it works fine again:

juju config openstack-dashboard enforce-ssl=false

The symptoms that I observed prior to applying the enforce-ssl workaround was that I would create an instance in Horizon and it would tell me the instance was created successfully but it wouldn't show up in the instances after that, nor in openstack server list. Also, sometimes I would not see Networks, Network Ports, Security Groups or Key Pair on the left in the instance creation window. I also could not create an object store container via Horizon. It would also say success but wouldn't actually create it.

[David Coronel (davecore)]

Subscribed ~field-high

[David Coronel (davecore)]

As an extra note, with enforce-ssl set to True in openstack-dashboard, I manually commented out the lines "Header edit Set-Cookie ^(.*)$ $1;HttpOnly;Secure" from /etc/apache2/sites-enabled/default-ssl.conf on my 3 openstack-dashboard units and I can create instances and object store containers successfully again.

ubuntu@juju-96d97f-21-lxd-14:~$ cat /etc/apache2/sites-enabled/default-ssl.conf
# Accept connections from non-SNI clients
SSLStrictSNIVHostCheck off
NameVirtualHost *:443
<VirtualHost 10.10.51.97:433>
    ServerName 10.10.51.30
[...]
    Header set Strict-Transport-Security "max-age=0"
    #Header edit Set-Cookie ^(.*)$ $1;HttpOnly;Secure

    Header set X-XSS-Protection "1; mode=block"
    Header set X-Content-Type-Options "nosniff"
[...]
</VirtualHost>

<VirtualHost 10.10.51.97:433>
    ServerName <fqdn hostname - redacted out>
[...]
    Header set Strict-Transport-Security "max-age=0"
    #Header edit Set-Cookie ^(.*)$ $1;HttpOnly;Secure

    Header set X-XSS-Protection "1; mode=block"
    Header set X-Content-Type-Options "nosniff"
[...]
</VirtualHost>

[Alex Kavanagh (ajkavanagh)]

Yes, okay, that's confirmed. The angular bits in horizon do now play well the "Set-Cookie" because Angular can't (in this instance) grab the CSRF cookie and pass it back to the server; thus the request isn't authenticated.

The charm needs a patch to take out the Set-Cookie (and release to stable), but also the original bug will be re-opened to determine a course of action.

I'll do this now.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/695918

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/19.10
Review: https://review.opendev.org/695920

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/695918
Committed: https://git.openstack.org/cgit/openstack/charm-openstack-dashboard/commit/?id=5375df0b12657e3494a966a48d04878aba8d153a
Submitter: Zuul
Branch: master

commit 5375df0b12657e3494a966a48d04878aba8d153a
Author: Alex Kavanagh <email address hidden>
Date: Mon Nov 25 13:05:40 2019 +0000

    Remove Set-Cookie .... HttpOnly;secure to allow CSRF access

    Angular (running in the page) can't access the CSRF token if the cookie
    is set to secure. This is a temporary patch to resolve the issue whilst
    a more permanent fix is found.

    This reverts patch I1ded951d79ad9fa832d1e88f656a1e064b1ef007
    (essentially).

    Change-Id: Id99abb429a0dc541ab5a3603962db8a563835eea
    Related-Bug: #1822751
    Closes-Bug: #1853173

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/695920
Committed: https://git.openstack.org/cgit/openstack/charm-openstack-dashboard/commit/?id=2ac72c39b480ae07df50edaac9802a7049773ca6
Submitter: Zuul
Branch: stable/19.10

commit 2ac72c39b480ae07df50edaac9802a7049773ca6
Author: Alex Kavanagh <email address hidden>
Date: Mon Nov 25 13:05:40 2019 +0000

    Remove Set-Cookie .... HttpOnly;secure to allow CSRF access

    Angular (running in the page) can't access the CSRF token if the cookie
    is set to secure. This is a temporary patch to resolve the issue whilst
    a more permanent fix is found.

    This reverts patch I1ded951d79ad9fa832d1e88f656a1e064b1ef007
    (essentially).

    Change-Id: Ied9d0f5486c260a17da9375ec6347d0952154225
    Related-Bug: #1822751
    Closes-Bug: #1853173