Fuel for OpenStack

[SSL] Horizon has been failed with an unexpected error has occurred when user tries to request horizon/project/access_and_security/

Bug #1479320 reported by Tatyanka on 2015-07-29

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Released	High	Michael Polenchuk	Fuel for OpenStack 7.0

Bug Description

Steps:
1. Deploy cluster
os: Ubuntu
neutron vlan
1 controller + 1 compute +1 cinder
ssl for os is enabled

2. As soon as cluster is ready, navigate to horizon over https
3. Login as admin user
4. Navigate to horizon/project/access_and_security/

Actual result:
Horizon became unavailable with message An unexpected error has occurred and 500 Internal error code
http://paste.openstack.org/show/406199/

at the same time logged user can list instances, images over horizon
[root@nailgun ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "103"
  build_id: "2015-07-28_12-51-16"
  nailgun_sha: "d7fe1047caeb4503970c7d39689e133b28b85b22"
  python-fuelclient_sha: "f04e6c46783ecd6000df31b61b6749da66d4d828"
  fuel-agent_sha: "2a65f11c10b0aeb5184247635a19740fc3edde21"
  fuel-nailgun-agent_sha: "1512b9af6b41cc95c4d891c593aeebe0faca5a63"
  astute_sha: "34e0493afa22999c4a07d3198ceb945116ab7932"
  fuel-library_sha: "8e64ae8e018d5b119c8e2ea49f6a83467b970a66"
  fuel-ostf_sha: "9e32b35e9a774eec41b86cd5a6c63b71a3efa4bd"
  fuelmain_sha: "0d6ff4c9ec156196cc4d9a36db314ba7346942f7"
[root@nailgun ~]# ssh node-1
Warning: Permanently added 'node-1' (RSA) to the list of known hosts.
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-59-generic x86_64)

Revision history for this message

Tatyanka (tatyana-leontovich) wrote on 2015-07-29:

fuel-snapshot-2015-07-29_11-37-15.tar.xz Edit (29.5 MiB, application/octet-stream)

Ihor Kalnytskyi (ikalnytskyi) on 2015-07-30

Changed in fuel:
status:	New → Confirmed

Stanislaw Bogatkin (sbogatkin) on 2015-07-30

Changed in fuel:
assignee:	Stanislaw Bogatkin (sbogatkin) → Tomasz Adam Jaroszewski (tjaroszewski)

Revision history for this message

Tomasz Adam Jaroszewski (tjaroszewski) wrote on 2015-08-05:

Download full text (4.5 KiB)

When HTTPS for Horizon is configured/enabled FUEL uses default hostname 'public.fuel.local' as:
a) CN for ssl certificate (generated using /etc/puppet/modules/osnailyfacter/modular/astute/generate_haproxy_keys.sh and executed by task `generate_haproxy_keys`)
b) Public endpoint for identify service.

/etc/puppet/2015.1.0-7.0/modules/osnailyfacter/modular/keystone/keystone.pp
41 $public_address = $public_ssl_hash['services'] ? {
>> 42 true => $public_ssl_hash['hostname'],
43 default => $public_vip,
44 }

Because SSL certificates are signed against hostnames and not IP addresses (and we have mixed public endpoints).. client is throwing "SSLError: hostname '__IP_ADDRESS!__' doesn't match u'public.fuel.local'.

HAProxy configuration:
/etc/haproxy/conf.d/015-horizon.cfg: bind 192.168.200.101:80
/etc/haproxy/conf.d/017-horizon-ssl.cfg: bind 192.168.200.101:443 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/020-keystone-1.cfg: bind 192.168.200.101:5000 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/040-nova-api-1.cfg: bind 192.168.200.101:8773 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/050-nova-api-2.cfg: bind 192.168.200.101:8774 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/070-cinder-api.cfg: bind 192.168.200.101:8776 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/080-glance-api.cfg: bind 192.168.200.101:9292 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/085-neutron.cfg: bind 192.168.200.101:9696 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/120-swift.cfg: bind 192.168.200.101:8080 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/160-heat-api.cfg: bind 192.168.200.101:8004 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/161-heat-api-cfn.cfg: bind 192.168.200.101:8000 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/162-heat-api-cloudwatch.cfg: bind 192.168.200.101:8003 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/170-nova-novncproxy.cfg: bind 192.168.200.101:6080 ssl crt /var/lib/astute/haproxy/public_haproxy.pem

To resolve this issue we could:

1) Use OPENSTACK_ENDPOINT_TYPE = "internalURL" in /etc/openstack-dashboard/local_settings.py
That way we instruct horizon to use only internalurl which don't use ssl.

/etc/puppet/2015.1.0-7.0/modules/osnailyfacter/modular/keystone/keystone.pp
41 $public_address  = $public_ssl_hash['services'] ? {
>> 42  true    => $public_ssl_hash['hostname'],
43  default => $public_vip,
44 }

HAProxy configuration:
/etc/haproxy/conf.d/015-horizon.cfg:  bind 192.168.200.101:80
/etc/haproxy/conf.d/017-horizon-ssl.cfg:  bind 192.168.200.101:443 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/020-keystone-1.cfg:  bind 192.168.200.101:5000 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/040-nova-api-1.cfg:  bind 192.168.200.101:8773 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/050-nova-api-2.cfg:  bind 192.168.200.101:8774 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/070-cinder-api.cfg:  bind 192.168.200.101:8776 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/080-glance-api.cfg:  bind 192.168.200.101:9292 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/085-neutron.cfg:  bind 192.168.200.101:9696 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/120-swift.cfg:  bind 192.168.200.101:8080 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/160-heat-api.cfg:  bind 192.168.200.101:8004 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/161-heat-api-cfn.cfg:  bind 192.168.200.101:8000 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/162-heat-api-cloudwatch.cfg:  bind 192.168.200.101:8003 ssl crt /var/lib/astute/haproxy/public_haproxy.pem
/etc/haproxy/conf.d/170-nova-novncproxy.cfg:  bind 192.168.200.101:6080 ssl crt /var/lib/astute/haproxy/public_haproxy.pem

To resolve this issue we could:

1) Use OPENSTACK_ENDPOINT_TYPE = "internalURL" in /etc/openstack-dashboard/local_settings.py
That way we instruct horizon to use only internalurl which don't use ssl.

2) In default configuration /etc/hosts entries were added (__PUB_IP__	public.fuel.local), currently this solution doesn't exactly works because of host_check.
To enable existing solution one would needs to:
1) Edit /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py and comment (lines) [621] and add [622]
621         #self.assert_hostname = assert_hostname
622         self.assert_hostname = False

There is also possibility to re-create certificates with IP as CN (which works but it is not compliant with standards - such information should be placed in extended attributes -> IP.1: __IP__):
a) Remove `$public_ssl_hash['hostname']..` from /etc/puppet/2015.1.0-7.0/modules/osnailyfacter/modular/keystone/keystone.pp . That way we can change (a) and (b) to use public_vip.
 -> `sed -i -e '/public_ssl_hash.*hostname.*/d' /etc/puppet/modules/osnailyfacter/modular/keystone/keystone.pp`
 and redeploy your environment
 
b) For existing environments one would need to:
 2a) update endpoints -> eg. `mysql> update endpoint set url='https://192.168.200.101:5000/v2.0' where url='https://public.fuel.local:5000/v2.0';`
 2b) Regenerate and redistribute cerificates:
 	vi /etc/puppet/modules/osnailyfacter/modular/astute/generate_haproxy_keys.sh 
 	Change SL_CN_NAME="$cn_name" to SL_CN_NAME="__IP_ADDRESS!__"
	
	[root@fuel modules]# fuel node --node 4 --tasks copy_haproxy_keys --env 2
	[root@fuel modules]# fuel node --node 4 --tasks generate_haproxy_keys --env 2
	Revert changes made to /etc/puppet/modules/osnailyfacter/modular/astute/generate_haproxy_keys.sh 
	
I am also checking if usage of certificate's extended attributes would be more elegant solution (if that will work as expected).

Aleksandr Didenko (adidenko) on 2015-08-11

Changed in fuel:
assignee:	Tomasz Adam Jaroszewski (tjaroszewski) → Fuel Library Team (fuel-library)
status:	Confirmed → Triaged

Michael Polenchuk (mpolenchuk) on 2015-08-11

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Michael Polenchuk (mpolenchuk)

Tatyanka (tatyana-leontovich) on 2015-08-11

summary:

- [SSL] Horizon has been failed with An unexpected error has occurred
+ [SSL] Horizon has been failed with an unexpected error has occurred
when user tries to request horizon/project/access_and_security/

Michael Polenchuk (mpolenchuk) on 2015-08-11

Changed in fuel:
status:	Triaged → In Progress

Revision history for this message

Michael Polenchuk (mpolenchuk) wrote on 2015-08-11:

related bug: https://bugs.launchpad.net/fuel/+bug/1478649

Revision history for this message

Stanislaw Bogatkin (sbogatkin) wrote on 2015-08-11:

Actually, I think that it is a duplicate, not just related. Also, after resolving this we could remove SSL_NO_VERIFY from Horizon, cause it will negotiate SSL right way.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-08-12: Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/211875

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-08-13: Related fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/209020
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=cd9c04c75c15b2bace7809fe219786b553e7fe88
Submitter: Jenkins
Branch: master

commit cd9c04c75c15b2bace7809fe219786b553e7fe88
Author: Stanislaw Bogatkin <email address hidden>
Date: Tue Aug 4 15:06:58 2015 +0300

Configure keystone v3 for nova

    Identity API v3 was added to kilo. This patch changes
    v3 endpoints from http://127.0.0.1:8774/v3 to actual proto://IP:Port/v3
    schema.

Co-Authored-By: Sergii Golovatiuk <email address hidden>

Related-Bug: #1479320
Closes-Bug: #1481296

Change-Id: I5ade31e5b30259cec76471ef932254e8e45cac22

OpenStack Infra (hudson-openstack) on 2015-08-13

Changed in fuel:
assignee:	Michael Polenchuk (mpolenchuk) → Sergii Golovatiuk (sgolovatiuk)

OpenStack Infra (hudson-openstack) on 2015-08-14

Changed in fuel:
assignee:	Sergii Golovatiuk (sgolovatiuk) → Michael Polenchuk (mpolenchuk)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-08-14: Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/211875
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=7cf18a6e29355f82bdc56b840f174a61ba0dff2d
Submitter: Jenkins
Branch: master

commit 7cf18a6e29355f82bdc56b840f174a61ba0dff2d
Author: Michael Polenchuk <email address hidden>
Date: Wed Aug 12 10:14:06 2015 +0300

Configure Swift S3 endpoints

- Configure Swift S3 endpoints

Closes-Bug: #1479320
Change-Id: Ic6e4850f1eec149d64281df4cc152deda271d86e

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

Alexander Kurenyshev (akurenyshev) wrote on 2015-09-04:

Verified on the #265 ISO.

Steps used to verify:

1) Install master node
2) Create cluster with 3 controllers, 2 computes and 1 cinder. Enables SSL for horizon
3) Deploy cluster
4) horizon/project/access_and_security/

Actual result:
Page is showed correctly without any errors. Security groups could be created/deleted