SSL not configured properly in reactive charms

Bug #1719331 reported by Tytus Kurek
30
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack AODH Charm
Fix Released
High
Liam Young
OpenStack Barbican Charm
Fix Released
High
Liam Young
OpenStack Designate Charm
Fix Released
High
Liam Young
charms.openstack
Fix Committed
High
Liam Young

Bug Description

This issue has been faced when deploying designate charm from openstack-charmers-next repository in the HA mode on Ubuntu Trusty with SSL charm options enabled. For some reason apache2 is not installed an the certificate / key files for the VIP are not being created:

root@juju-64cbea-22-lxd-26:~# service apache2 status
apache2: unrecognized service

root@juju-64cbea-22-lxd-26:~# ls -l /etc/apache2/ssl/designate/
total 8
-r--r--r-- 1 root root 1176 Sep 25 11:00 cert_100.86.0.185
-r--r--r-- 1 root root 1679 Sep 25 11:00 key_100.86.0.185

This can be worked around by executing the following commands:

$ apt -y install apache2
$ a2ensite openstack_https_frontend.conf
$ a2enmod ssl
$ a2enmod proxy
$ cp /etc/apache2/ssl/designate/cert_100.86.0.* /etc/apache2/ssl/designate/cert_100.86.0.12
$ cp /etc/apache2/ssl/designate/key_100.86.0.* /etc/apache2/ssl/designate/key_100.86.0.12
$ service apache2 start

Designate charm is configured as follows:

application: designate
charm: designate
settings:
  debug:
    description: Enable debug logging
    type: boolean
    value: true
  dns-slaves:
    default: true
    description: |
      List of DNS slaves which will accept addzone/delzone rndc commands from
      Designate. List is of the form slave_ip:rndc_port:rndc_key. This should
      only be used if DNS servers are outside of Juju control. Using the
      designate-bind charm is the prefered approach.
    type: string
  enable-host-header:
    description: Enables host request headers.
    type: boolean
    value: true
  haproxy-client-timeout:
    default: true
    description: |
      Client timeout configuration in ms for haproxy, used in HA
      configurations.
    type: int
    value: 30000
  haproxy-connect-timeout:
    default: true
    description: |
      Connect timeout configuration in ms for haproxy, used in HA
      configurations.
    type: int
    value: 5000
  haproxy-queue-timeout:
    default: true
    description: |
      Queue timeout configuration in ms for haproxy, used in HA
      configurations.
    type: int
    value: 5000
  haproxy-server-timeout:
    default: true
    description: |
      Server timeout configuration in ms for haproxy, used in HA
      configurations.
    type: int
    value: 30000
  nameservers:
    description: |
      Space delimited list of nameservers. These are the nameservers that have
      been provided to the domain registrar in order to delegate the domain to
      Designate. e.g. "ns1.example.com. ns2.example.com."
    type: string
    value: <list of nameservers>
  neutron-domain:
    default: true
    description: Domain to add floating IP records to.
    type: string
  neutron-domain-email:
    default: true
    description: Email address of the person responsible for the domain.
    type: string
  neutron-record-format:
    default: true
    description: Format of floating IP global records.
    type: string
    value: '%(octet0)s-%(octet1)s-%(octet2)s-%(octet3)s.%(zone)s'
  nova-domain:
    default: true
    description: Domain to add records for new instances to
    type: string
  nova-domain-email:
    default: true
    description: Email address of the person responsible for the domain.
    type: string
  nova-record-format:
    default: true
    description: Format of floating IP global records.
    type: string
    value: '%(hostname)s.%(tenant_id)s.%(zone)s'
  openstack-origin:
    description: |
      Repository from which to install OpenStack.

      May be one of the following:

        distro (default)
        ppa:somecustom/ppa (PPA name must include OpenStack Release)
        deb url sources entry|key id
        or a supported Ubuntu Cloud Archive pocket.

      Supported Ubuntu Cloud Archive pockets include:

        cloud:trusty-liberty
        cloud:trusty-juno
        cloud:trusty-kilo
        cloud:trusty-liberty
        cloud:trusty-mitaka

      Note that updating this setting to a source that is known to
      provide a later version of OpenStack will trigger a software
      upgrade.
    type: string
    value: cloud:trusty-mitaka
  os-admin-hostname:
    default: true
    description: |
      The hostname or address of the admin endpoints created in the keystone
      identity provider.
      .
      This value will be used for admin endpoints. For example, an
      os-admin-hostname set to 'api-admin.example.com' with ssl enabled
      will create the following endpoint for neutron-api:
      .
      https://api-admin.example.com:9696/
    type: string
  os-admin-network:
    default: true
    description: |
      The IP address and netmask of the OpenStack Admin network (e.g.,
      192.168.0.0/24)
      .
      This network will be used for admin endpoints.
    type: string
  os-internal-hostname:
    default: true
    description: |
      The hostname or address of the internal endpoints created in the keystone
      identity provider.
      .
      This value will be used for internal endpoints. For example, an
      os-internal-hostname set to 'api-internal.example.com' with ssl enabled
      will create the following endpoint for neutron-api:
      .
      https://api-internal.example.com:9696/
    type: string
  os-internal-network:
    default: true
    description: |
      The IP address and netmask of the OpenStack Internal network (e.g.,
      192.168.0.0/24)
      .
      This network will be used for internal endpoints.
    type: string
  os-public-hostname:
    default: true
    description: |
      The hostname or address of the public endpoints created in the keystone
      identity provider.
      .
      This value will be used for public endpoints. For example, an
      os-public-hostname set to 'api-public.example.com' with ssl enabled
      will create the following endpoint for neutron-api:
      .
      https://api-public.example.com:9696/
    type: string
  os-public-network:
    default: true
    description: |
      The IP address and netmask of the OpenStack Public network (e.g.,
      192.168.0.0/24)
      .
      This network will be used for public endpoints.
    type: string
  region:
    default: true
    description: OpenStack Region
    type: string
    value: RegionOne
  ssl_ca:
    description: |
      SSL CA to use with the certificate and key provided - this is only
      required if you are providing a privately signed ssl_cert and ssl_key.
    type: string
    value: |-
      <CA certificate>
  ssl_cert:
    description: |
      SSL certificate to install and use for API ports. Setting this value
      and ssl_key will enable reverse proxying, point Glance's entry in the
      Keystone catalog to use https, and override any certficiate and key
      issued by Keystone (if it is configured to do so).
    type: string
    value: |-
      <Designate certificate>
  ssl_key:
    description: |
      SSL key to use with certificate specified as ssl_cert.
    type: string
    value: |-
      <Designate certificate key>
  use-internal-endpoints:
    default: true
    description: |
      Openstack mostly defaults to using public endpoints for
      internal communication between services. If set to True this option
      will configure services to use internal endpoints where possible.
    type: boolean
    value: false
  use-syslog:
    description: |
      Setting this to True will allow supporting services to log to syslog.
    type: boolean
    value: true
  verbose:
    description: Enable verbose logging
    type: boolean
    value: true
  vip:
    description: |
      Virtual IP(s) to use to front API services in HA configuration.

      If multiple networks are being used, a VIP should be provided for each
      network, separated by spaces.
    type: string
    value: 100.86.0.12
  vip_cidr:
    default: true
    description: |
      Default CIDR netmask to use for HA vip when it cannot be automatically
      determined.
    type: int
    value: 24
  vip_iface:
    default: true
    description: |
      Default network interface to use for HA vip when it cannot be
      automatically determined.
    type: string
    value: eth0
  worker-multiplier:
    default: true
    description: |
      The CPU core multiplier to use when configuring worker processes. By
      default, the number of workers for each daemon is set to twice the number
      of CPU cores a service unit has. When deployed in a LXD container, this
      default value will be capped to 4 workers unless this configuration
      option is set.
    type: float

Attached are the logs from all 3 units.

Revision history for this message
Tytus Kurek (tkurek) wrote :
tags: added: 4010 cpe-onsite
Revision history for this message
Nobuto Murata (nobuto) wrote :

FWIW, I'm not 100% sure, but this might have a connection with:
https://bugs.launchpad.net/charm-aodh/+bug/1713944
since both are reactive OpenStack charms.

Liam Young (gnuoy)
Changed in charm-designate:
assignee: nobody → Liam Young (gnuoy)
Liam Young (gnuoy)
Changed in charm-designate:
status: New → Confirmed
importance: Undecided → High
Revision history for this message
Pen Gale (pengale) wrote :

I don't think that this is related to https://bugs.launchpad.net/charm-aodh/+bug/1713944

That bug is related to charmhelpers writing incorrect filenames into the apache2 config, possibly due to a broken dns in the deployment.

In this one, the values simply aren't being written, and the only Traceback I see in the logs seems to be only tangentially related :-/

Liam Young (gnuoy)
summary: - apache2 not installed when using SSL options
+ SSL not configured properly in reactive charms
Changed in charm-barbican:
status: New → Confirmed
Changed in charm-aodh:
status: New → Confirmed
assignee: nobody → Liam Young (gnuoy)
Changed in charm-barbican:
assignee: nobody → Liam Young (gnuoy)
importance: Undecided → High
Changed in charm-aodh:
importance: Undecided → High
Liam Young (gnuoy)
Changed in charms.openstack:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Liam Young (gnuoy)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-aodh (master)

Fix proposed to branch: master
Review: https://review.openstack.org/507977

Changed in charm-aodh:
status: Confirmed → In Progress
Changed in charm-designate:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-designate (master)

Fix proposed to branch: master
Review: https://review.openstack.org/507978

Changed in charm-barbican:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-barbican (master)

Fix proposed to branch: master
Review: https://review.openstack.org/507979

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-designate (master)

Reviewed: https://review.openstack.org/507978
Committed: https://git.openstack.org/cgit/openstack/charm-designate/commit/?id=a2375ca7f2fe6a7c67ebf48f9fe51f4ae1445dc0
Submitter: Zuul
Branch: master

commit a2375ca7f2fe6a7c67ebf48f9fe51f4ae1445dc0
Author: Liam Young <email address hidden>
Date: Wed Sep 27 20:38:35 2017 +0000

    Rebuild charm to get SSL fixes

    Trigger a rebuild of the charm to pickup an update to
    charms.openstack which fixes enabling SSL for API
    endpoint.

    Change-Id: I6e5deab2d1f7e5d98ec4b29762ebeebe601ad786
    Partial-Bug: #1719331

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-barbican (master)

Reviewed: https://review.openstack.org/507979
Committed: https://git.openstack.org/cgit/openstack/charm-barbican/commit/?id=d2a4ec2bc92b2d6413f4af05d11987e83bc21740
Submitter: Zuul
Branch: master

commit d2a4ec2bc92b2d6413f4af05d11987e83bc21740
Author: Liam Young <email address hidden>
Date: Wed Sep 27 20:34:20 2017 +0000

    Rebuild charm to get SSL fixes

    Trigger a rebuild of the charm to pickup an update to
    charms.openstack which fixes enabling SSL for API
    endpoint.

    Change-Id: If5db1d72da041ed07159ab5af8fe41a863c92549
    Partial-Bug: #1719331

Ryan Beisner (1chb1n)
Changed in charm-aodh:
milestone: none → 17.11
Changed in charm-barbican:
milestone: none → 17.11
Changed in charm-designate:
milestone: none → 17.11
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-aodh (master)

Reviewed: https://review.openstack.org/507977
Committed: https://git.openstack.org/cgit/openstack/charm-aodh/commit/?id=ac98ae51dcdd79d604b4dbbdd35388f513ea17ea
Submitter: Zuul
Branch: master

commit ac98ae51dcdd79d604b4dbbdd35388f513ea17ea
Author: Liam Young <email address hidden>
Date: Wed Sep 27 20:39:13 2017 +0000

    Rebuild charm to get SSL fixes

    Trigger a rebuild of the charm to pickup an update to
    charms.openstack which fixes enabling SSL for API
    endpoint.

    Change-Id: I681da6617a1026ca9b6c7f13ac2ce70ad52c912a
    Partial-Bug: #1719331

Liam Young (gnuoy)
Changed in charms.openstack:
status: Confirmed → Fix Committed
Changed in charm-designate:
status: In Progress → Fix Committed
Changed in charm-barbican:
status: In Progress → Fix Committed
Changed in charm-aodh:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-aodh (stable/17.08)

Fix proposed to branch: stable/17.08
Review: https://review.openstack.org/514980

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-aodh (stable/17.08)

Reviewed: https://review.openstack.org/514980
Committed: https://git.openstack.org/cgit/openstack/charm-aodh/commit/?id=56b02097b4fb4066b2552fab135caefac061086e
Submitter: Zuul
Branch: stable/17.08

commit 56b02097b4fb4066b2552fab135caefac061086e
Author: Dmitrii Shcherbakov <email address hidden>
Date: Wed Oct 25 11:28:10 2017 +0200

    Rebuild charm to get SSL fixes

    Trigger a rebuild of the charm to pickup an update to
    charms.openstack which fixes enabling SSL for API endpoint.

    Change-Id: I27d334c3e5ed445e8d8d0ba2d26d115f6d5992c1
    Partial-Bug: #1719331

James Page (james-page)
Changed in charm-designate:
status: Fix Committed → Fix Released
Changed in charm-aodh:
status: Fix Committed → Fix Released
Changed in charm-barbican:
status: Fix Committed → Fix Released
Revision history for this message
Michael Skalka (mskalka) wrote :

As of 2018-1-8 this bug persists for the aodh charm.

Similar symptoms as originally reported, apache2 is installing correctly however the certificates for the VIPs were still missing in /etc/apache2/ssl/aodh/.

As before, copying the other certs to match the VIP IPs resolved the issue:

$ cp /etc/apache2/ssl/aodh/[key, cert]_X.X.X.* /etc/apache2/ssl/aodh/[key, cert]_[VIP-1, VIP-2]
$ systemctl restart apache2

Revision history for this message
Liam Young (gnuoy) wrote :

I have been unable to reproduce this issue using the current stable version of aodh ( cs:aodh-10 ).

⟫ aodh capabilities list --insecure
+---------------+----------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------+----------------------------------------------------------------------------------------------------------------------------------------+
| alarm_storage | {'storage:production_ready': True} |
| api | {'alarms:query:simple': True, 'alarms:history:query:complex': True, 'alarms:query:complex': True, 'alarms:history:query:simple': True} |
+---------------+----------------------------------------------------------------------------------------------------------------------------------------+

Aodh was deployed using:

applications:
  keystone:
    charm: cs:keystone
    num_units: 1
    options:
      admin-password: openstack
      admin-token: ubuntutesting
      https-service-endpoints: "True"
      use-https: "yes"
      openstack-origin: cloud:xenial-pike
      worker-multiplier: 0.25
  mysql:
    charm: cs:percona-cluster
    num_units: 1
    options:
    options:
      innodb-buffer-pool-size: 256M
      max-connections: 1000
  rabbitmq-server:
    charm: cs:rabbitmq-server
    num_units: 1
    options:
      ssl: "only"
  aodh:
    charm: aodh
    num_units: 1
    constraints: mem=1G
relations:
  - [ keystone, mysql ]
  - [ aodh, rabbitmq-server ]
  - [ aodh, mysql ]
  - [ aodh, keystone ]

Revision history for this message
Michael Skalka (mskalka) wrote :

Aodh/Keystone versions:

App Version Status Scale Charm Store Rev OS Notes
aodh 5.0.0 active 3 aodh jujucharms 10 ubuntu
hacluster-aodh active 3 hacluster jujucharms 38 ubuntu
hacluster-keystone active 3 hacluster jujucharms 38 ubuntu
keystone 12.0.0 active 3 keystone jujucharms 272 ubuntu

Config:
variables:
  ...
  openstack-origin: &openstack-origin cloud:xenial-pike
  ...
services:
  aodh:
    charm: cs:xenial/aodh
    num_units: 3
    bindings:
      "": *oam-space
      public: *public-space
      admin: *admin-space
      internal: *internal-space
      shared-db: *internal-space
    options:
      openstack-origin: *openstack-origin
      region: *openstack-region
      vip: *aodh-vip
      os-admin-network: *os-admin-network
      os-public-network: *os-public-network
      os-internal-network: *os-internal-network
      ssl_ca: *ssl_ca
      ssl_cert: *ssl_cert
      ssl_key: *ssl_key
      os-public-hostname: *hostname_aodh_public
      os-internal-hostname: *hostname_aodh_internal
      os-admin-hostname: *hostname_aodh_internal
  ...

  ...
  keystone:
    charm: cs:xenial/keystone
    num_units: 3
    bindings:
      "": *oam-space
      public: *public-space
      admin: *admin-space
      internal: *internal-space
      shared-db: *internal-space
    options:
      worker-multiplier: *worker-multiplier
      openstack-origin: *openstack-origin
      vip: *keystone-vip
      region: *openstack-region
      preferred-api-version: 3
      #admin-password: openstack
      os-admin-network: *os-admin-network
      os-public-network: *os-public-network
      os-internal-network: *os-internal-network
      ssl_ca: *ssl_ca
      ssl_cert: *ssl_cert
      ssl_key: *ssl_key
      os-public-hostname: *hostname_keystone_public
      os-internal-hostname: *hostname_keystone_internal
      os-admin-hostname: *hostname_keystone_internal

Revision history for this message
Liam Young (gnuoy) wrote :

Hi Michael,
    I was able to reproduce this. The issue is very slightly different to this bug so I've raised a new bug to cover further investigation and any potential fix Bug #1744886
Liam

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.