[ssl + hardening apache trusty-mitaka] caught SIGTERM, shutting down on juju update hook.

[José Pekkarinen]

On a Trusty mitaka openstack deployment where dashboard is deployed on HA, every juju update hook
I can read the following output on /var/log/apache2/error.log:

[Mon Feb 05 12:21:57.599471 2018] [core:notice] [pid 45036:tid 140322761824128] AH00094: Command line: '/usr/sbin/apache2'
[Mon Feb 05 12:27:15.038971 2018] [mpm_event:notice] [pid 45036:tid 140322761824128] AH00491: caught SIGTERM, shutting down
[Mon Feb 05 12:27:16.133030 2018] [ssl:warn] [pid 1116:tid 139686050834304] AH01909: juju-baab69-29-lxd-2.maas:433:0 server certificate does NOT include an ID which matches the server name
[Mon Feb 05 12:27:16.149425 2018] [ssl:warn] [pid 1117:tid 139686050834304] AH01909: juju-baab69-29-lxd-2.maas:433:0 server certificate does NOT include an ID which matches the server name
[Mon Feb 05 12:27:16.151705 2018] [mpm_event:notice] [pid 1117:tid 139686050834304] AH00489: Apache/2.4.10 (Ubuntu) OpenSSL/1.0.1f mod_wsgi/3.4 Python/2.7.6 configured -- resuming normal operations
[Mon Feb 05 12:27:16.151729 2018] [core:notice] [pid 1117:tid 139686050834304] AH00094: Command line: '/usr/sbin/apache2'
[Mon Feb 05 12:31:49.834311 2018] [mpm_event:notice] [pid 1117:tid 139686050834304] AH00491: caught SIGTERM, shutting down
[Mon Feb 05 12:31:50.931838 2018] [ssl:warn] [pid 6030:tid 139668206278528] AH01909: juju-baab69-29-lxd-2.maas:433:0 server certificate does NOT include an ID which matches the server name
[Mon Feb 05 12:31:50.947896 2018] [ssl:warn] [pid 6031:tid 139668206278528] AH01909: juju-baab69-29-lxd-2.maas:433:0 server certificate does NOT include an ID which matches the server name
[Mon Feb 05 12:31:50.950319 2018] [mpm_event:notice] [pid 6031:tid 139668206278528] AH00489: Apache/2.4.10 (Ubuntu) OpenSSL/1.0.1f mod_wsgi/3.4 Python/2.7.6 configured -- resuming normal operations
[Mon Feb 05 12:31:50.950349 2018] [core:notice] [pid 6031:tid 139668206278528] AH00094: Command line: '/usr/sbin/apache2'
[Mon Feb 05 12:37:45.785726 2018] [mpm_event:notice] [pid 6031:tid 139668206278528] AH00491: caught SIGTERM, shutting down
[Mon Feb 05 12:37:46.882574 2018] [ssl:warn] [pid 11545:tid 140014746675072] AH01909: juju-baab69-29-lxd-2.maas:433:0 server certificate does NOT include an ID which matches the server name
[Mon Feb 05 12:37:46.898069 2018] [ssl:warn] [pid 11546:tid 140014746675072] AH01909: juju-baab69-29-lxd-2.maas:433:0 server certificate does NOT include an ID which matches the server name
[Mon Feb 05 12:37:46.900170 2018] [mpm_event:notice] [pid 11546:tid 140014746675072] AH00489: Apache/2.4.10 (Ubuntu) OpenSSL/1.0.1f mod_wsgi/3.4 Python/2.7.6 configured -- resuming normal operations
[Mon Feb 05 12:37:46.900195 2018] [core:notice] [pid 11546:tid 140014746675072] AH00094: Command line: '/usr/sbin/apache2'
[Mon Feb 05 12:43:00.907437 2018] [mpm_event:notice] [pid 11546:tid 140014746675072] AH00491: caught SIGTERM, shutting down
[Mon Feb 05 12:43:02.002314 2018] [ssl:warn] [pid 16737:tid 140561918715776] AH01909: juju-baab69-29-lxd-2.maas:433:0 server certificate does NOT include an ID which matches the server name
[Mon Feb 05 12:43:02.017202 2018] [ssl:warn] [pid 16738:tid 140561918715776] AH01909: juju-baab69-29-lxd-2.maas:433:0 server certificate does NOT include an ID which matches the server name
[Mon Feb 05 12:43:02.019455 2018] [mpm_event:notice] [pid 16738:tid 140561918715776] AH00489: Apache/2.4.10 (Ubuntu) OpenSSL/1.0.1f mod_wsgi/3.4 Python/2.7.6 configured -- resuming normal operations
[Mon Feb 05 12:43:02.019479 2018] [core:notice] [pid 16738:tid 140561918715776] AH00094: Command line: '/usr/sbin/apache2'

Also, but not all the update hooks, it's possible to see the following output:

[Mon Feb 05 13:08:32.034604 2018] [:error] [pid 34010] [remote 100.86.0.145:54985] mod_wsgi (pid=34010): Exception occurred processing WSGI script '/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi'.
[Mon Feb 05 13:08:32.034682 2018] [:error] [pid 34010] [remote 100.86.0.145:54985] IOError: failed to write data

The configuration looks like:

$ juju config openstack-dashboard
application: openstack-dashboard
charm: openstack-dashboard
settings:
  action-managed-upgrade:
    default: true
    description: |
      If True enables openstack upgrades for this charm via juju actions.
      You will still need to set openstack-origin to the new repository but
      instead of an upgrade running automatically across all units, it will
      wait for you to execute the openstack-upgrade action for this charm on
      each unit. If False it will revert to existing behavior of upgrading
      all units on config change.
    type: boolean
    value: false
  cinder-backup:
    description: Enable cinder backup panel.
    type: boolean
    value: true
  customization-module:
    default: true
    description: |
      Module that overriding layout for customization.
      This is available from Liberty
    type: string
    value: ""
  database:
    default: true
    description: Database name for Horizon (if enabled)
    type: string
    value: horizon
  database-user:
    default: true
    description: Username for Horizon database access (if enabled)
    type: string
    value: horizon
  debug:
    description: Show Django debug messages.
    type: string
    value: "false"
  default-role:
    default: true
    description: |
      Default role for Horizon operations that will be created in
      Keystone upon introduction of an identity-service relation.
    type: string
    value: Member
  default-theme:
    default: true
    description: |
      Specify path to theme to use
      (relative to /usr/share/openstack-dashboard/openstack_dashboard/themes/).
      NOTE: This setting is supported when deploying OpenStack Liberty or
      newer. This setting and ubuntu-theme setting are mutually exclusive.
    type: string
  dns-ha:
    default: true
    description: |
      Use DNS HA with MAAS 2.0. Note if this is set do not set vip
      settings below.
    type: boolean
    value: false
  endpoint-type:
    description: |
      Specifies the endpoint types to use for endpoints in the Keystone
      service catalog. Valid values are 'publicURL', 'internalURL',
      and 'adminURL'. Both the primary and secondary endpoint types can
      be specified by providing multiple comma delimited values.
    type: string
    value: publicURL
  enforce-ssl:
    default: true
    description: |
      If True, redirects plain http requests to https port 443. For this option
      to have an effect, SSL must be configured.
    type: boolean
    value: false
  ha-bindiface:
    default: true
    description: |
      Default network interface on which HA cluster will bind to communication
      with the other members of the HA Cluster.
    type: string
    value: ethX
  ha-mcastport:
    default: true
    description: |
      Default multicast port number that will be used to communicate between
      HA Cluster nodes.
    type: int
    value: 5410
  haproxy-client-timeout:
    default: true
    description: |
      Client timeout configuration in ms for haproxy, used in HA
      configurations. If not provided, default value of 30000ms is used.
    type: int
  haproxy-connect-timeout:
    default: true
    description: |
      Connect timeout configuration in ms for haproxy, used in HA
      configurations. If not provided, default value of 5000ms is used.
    type: int
  haproxy-queue-timeout:
    default: true
    description: |
      Queue timeout configuration in ms for haproxy, used in HA
      configurations. If not provided, default value of 5000ms is used.
    type: int
  haproxy-server-timeout:
    default: true
    description: |
      Server timeout configuration in ms for haproxy, used in HA
      configurations. If not provided, default value of 30000ms is used.
    type: int
  harden:
    description: |
      Apply system hardening. Supports a space-delimited list of modules
      to run. Supported modules currently include os, ssh, apache and mysql.
    type: string
    value: apache
  nagios_check_http_params:
    default: true
    description: Parameters to pass to the nrpe plugin check_http.
    type: string
    value: -H localhost -I 127.0.0.1 -u '/' -e 200,301,302
  nagios_context:
    default: true
    description: |
      Used by the nrpe-external-master subordinate charm.
      A string that will be prepended to instance name to set the host name
      in nagios. So for instance the hostname would be something like:
      .
        juju-postgresql-0
      .
      If you're running multiple environments with the same services in them
      this allows you to differentiate between them.
    type: string
    value: juju
  nagios_servicegroups:
    default: true
    description: |
      A comma-separated list of nagios servicegroups. If left empty, the
      nagios_context will be used as the servicegroup.
    type: string
    value: ""
  neutron-network-dvr:
    default: true
    description: |
      Enable Neutron distributed virtual router (DVR) feature in the
      Router panel.
    type: boolean
    value: false
  neutron-network-firewall:
    default: true
    description: Enable neutron firewall service panel.
    type: boolean
    value: false
  neutron-network-l3ha:
    default: true
    description: |
      Enable HA (High Availability) mode in Neutron virtual router in
      the Router panel.
    type: boolean
    value: false
  neutron-network-lb:
    default: true
    description: Enable neutron load balancer service panel.
    type: boolean
    value: false
  neutron-network-vpn:
    default: true
    description: Enable neutron vpn service panel.
    type: boolean
    value: false
  offline-compression:
    default: true
    description: Use pre-generated Less compiled JS and CSS.
    type: string
    value: "yes"
  openstack-origin:
    description: |
      Repository from which to install. May be one of the following:
      distro (default), ppa:somecustom/ppa, a deb url sources entry,
      or a supported Cloud Archive release pocket.

      Supported Cloud Archive sources include:

      cloud:<series>-<openstack-release>
      cloud:<series>-<openstack-release>/updates
      cloud:<series>-<openstack-release>/staging
      cloud:<series>-<openstack-release>/proposed

      For series=Precise we support cloud archives for openstack-release:
         * icehouse

      For series=Trusty we support cloud archives for openstack-release:
         * juno
         * kilo
         * ...

      NOTE: updating this setting to a source that is known to provide
      a later version of OpenStack will trigger a software upgrade.
    type: string
    value: cloud:trusty-mitaka/updates
  openstack-origin-git:
    default: true
    description: |
      Specifies a default OpenStack release name, or a YAML dictionary
      listing the git repositories to install from.

      The default Openstack release name may be one of the following, where
      the corresponding OpenStack github branch will be used:
        * liberty
        * mitaka
        * newton
        * master

      The YAML must minimally include requirements and horizon repositories,
      and may also include repositories for other dependencies:
        repositories:
        - {name: requirements,
           repository: 'git://github.com/openstack/requirements',
           branch: master}
        - {name: horizon,
           repository: 'git://github.com/openstack/horizon',
           branch: master}
        release: master
    type: string
  os-admin-hostname:
    default: true
    description: |
      The hostname or address of the admin endpoints created for
      openstack-dashboard.

      This value will be used for admin endpoints. For example, an
      os-admin-hostname set to 'horizon.admin.example.com' with will create
      the following admin endpoint for the swift-proxy:

      https://horizon.admin.example.com/horizon
    type: string
  os-admin-network:
    description: |
      The IP address and netmask of the OpenStack Admin network (e.g., 192.168.0.0/24).
      This network will be used for admin endpoints.
    type: string
    value: X.X.X.X/Y
  os-internal-hostname:
    default: true
    description: |
      The hostname or address of the internal endpoints created for
      openstack-dashboard.

      This value will be used for internal endpoints. For example, an
      os-internal-hostname set to 'horizon.internal.example.com' with will
      create the following internal endpoint for the swift-proxy:

      https://horizon.internal.example.com/horizon
    type: string
  os-internal-network:
    default: true
    description: |
      The IP address and netmask of the OpenStack Internal network (e.g., 192.168.0.0/24).
      This network will be used for internal endpoints.
    type: string
  os-public-hostname:
    default: true
    description: |
      The hostname or address of the public endpoints created for
      openstack-dashboard.

      This value will be used for public endpoints. For example, an
      os-public-hostname set to 'horizon.example.com' with will create
      the following public endpoint for the swift-proxy:

      https://horizon.example.com/horizon
    type: string
  os-public-network:
    default: true
    description: |
      The IP address and netmask of the OpenStack Public network (e.g., 192.168.0.0/24).
      This network will be used for public endpoints.
    type: string
  password-retrieve:
    description: Enable "Retrieve password" instance action.
    type: boolean
    value: true
  prefer-ipv6:
    default: true
    description: |
      If True enables IPv6 support. The charm will expect network
      interfaces to be configured with an IPv6 address. If set to False
      (default) IPv4 is expected.
      .
      NOTE: these charms do not currently support IPv6 privacy extension.
      In order for this charm to function correctly, the privacy extension
      must be disabled and a non-temporary address must be
      configured/available on your network interface.
    type: boolean
    value: false
  profile:
    default: true
    description: Default profile for the dashboard. Eg. cisco.
    type: string
  secret:
    description: |
      Secret for Horizon to use when securing internal data; set this when
      using multiple dashboard units.
    type: string
    value: <valid_secret>
  ssl_ca:
    description: |
      Base64-encoded certificate authority. This CA is used in conjunction
      with keystone https endpoints and must, therefore, be the same CA
      used by any endpoint configured as https/ssl.
    type: string
    value: |-
      <my_key_here>
  ssl_cert:
    description: |
      Base64-encoded SSL certificate to install and use for Horizon.

       juju set openstack-dashbaord ssl_cert="$(cat cert| base64)" \
                                    ssl_key="$(cat key| base64)"
    type: string
    value: |-
      <my_key_here>
  ssl_key:
    description: |
      Base64-encoded SSL key to use with certificate specified as ssl_cert.
    type: string
    value: |-
      <my_key_here>
  ubuntu-theme:
    default: true
    description: Use Ubuntu theme for the dashboard.
    type: string
    value: "yes"
  use-syslog:
    description: |
      Setting this to True will allow supporting services to log to syslog.
    type: boolean
    value: true
  vip:
    description: |
      Virtual IP to use to front openstack dashboard ha configuration.
    type: string
    value: X.X.X.X
  vip_cidr:
    default: true
    description: |
      Default CIDR netmask to use for HA vip when it cannot be automatically
      determined.
    type: int
    value: 24
  vip_iface:
    default: true
    description: |
      Default network interface to use for HA vip when it cannot be
      automatically determined.
    type: string
    value: ethY
  webroot:
    description: |
      Directory where application will be accessible, relative to
      http://$hostname/.
    type: string
    value: /

The cycle shown from the juju unit log:

2018-02-05 16:32:20 DEBUG update-status * apache2 is running
2018-02-05 16:32:20 INFO juju-log Unit is ready
2018-02-05 16:32:22 INFO juju.worker.uniter.operation runhook.go:113 ran "update-status" hook
2018-02-05 16:36:38 INFO juju-log Registered config file: /usr/share/openstack-dashboard/openstack_dashboard/conf/keystonev3_policy.json
2018-02-05 16:36:38 INFO juju-log Registered config file: /etc/openstack-dashboard/local_settings.py
2018-02-05 16:36:38 INFO juju-log Registered config file: /etc/haproxy/haproxy.cfg
2018-02-05 16:36:38 INFO juju-log Registered config file: /etc/apache2/ports.conf
2018-02-05 16:36:40 INFO juju-log Registered config file: /etc/apache2/sites-available/000-default.conf
2018-02-05 16:36:40 INFO juju-log Registered config file: /etc/apache2/conf-available/openstack-dashboard.conf
2018-02-05 16:36:40 INFO juju-log Registered config file: /etc/apache2/sites-available/default-ssl.conf
2018-02-05 16:36:40 INFO juju-log Registered config file: /usr/share/openstack-dashboard/openstack_dashboard/enabled/_40_router.py
2018-02-05 16:36:40 DEBUG juju-log Hardening function 'install'
2018-02-05 16:36:40 DEBUG juju-log Hardening function 'upgrade_charm'
2018-02-05 16:36:40 DEBUG juju-log Hardening function 'config_changed'
2018-02-05 16:36:40 DEBUG juju-log Hardening function 'update_status'
2018-02-05 16:36:40 DEBUG juju-log Executing hardening module 'run_apache_checks'
2018-02-05 16:36:40 DEBUG juju-log Starting Apache hardening checks.
2018-02-05 16:36:40 DEBUG juju-log Found user-provided config overrides file '/var/lib/juju/agents/unit-openstack-dashboard-0/charm/hardening.yaml'
2018-02-05 16:36:40 DEBUG juju-log No overrides found for 'apache'
2018-02-05 16:36:40 DEBUG juju-log Running 'FilePermissionAudit' check
2018-02-05 16:36:40 DEBUG juju-log Running 'TemplatedFile' check
2018-02-05 16:36:40 INFO juju-log File /etc/apache2/mods-available/alias.conf has incorrect permissions, currently set to 0751
2018-02-05 16:36:40 INFO juju-log File /etc/apache2/mods-available/alias.conf is not in compliance.
2018-02-05 16:36:40 INFO juju-log Applying compliance criteria to '/etc/apache2/mods-available/alias.conf'
2018-02-05 16:36:40 DEBUG juju-log Rendering from template: alias.conf
2018-02-05 16:36:40 DEBUG juju-log Wrote template /etc/apache2/mods-available/alias.conf
2018-02-05 16:36:40 DEBUG juju-log Running service 'apache2' actions '['restart']'
2018-02-05 16:36:40 DEBUG update-status * Restarting web server apache2
2018-02-05 16:36:42 DEBUG update-status ...done.
2018-02-05 16:36:42 DEBUG juju-log Running 'TemplatedFile' check
2018-02-05 16:36:42 DEBUG juju-log Running 'DirectoryPermissionAudit' check
2018-02-05 16:36:43 DEBUG juju-log Running 'DisabledModuleAudit' check
2018-02-05 16:36:43 DEBUG juju-log Running 'NoReadWriteForOther' check
2018-02-05 16:36:43 INFO juju-log File /etc/apache2 is not in compliance.
2018-02-05 16:36:43 INFO juju-log Applying compliance criteria to '/etc/apache2'
2018-02-05 16:36:43 DEBUG juju-log Apache hardening checks complete.
2018-02-05 16:36:43 INFO juju-log Updating status.
2018-02-05 16:36:44 INFO juju-log Generating template context for identity-service
2018-02-05 16:36:44 INFO juju-log Missing required data: service_port service_host
2018-02-05 16:36:44 DEBUG update-status ERROR no relation id specified
2018-02-05 16:36:44 INFO juju-log CA cert is the same as installed version
2018-02-05 16:36:44 INFO juju-log Generating template context for identity-service
2018-02-05 16:36:44 INFO juju-log Missing required data: service_port service_host
2018-02-05 16:36:44 INFO juju-log Ensuring haproxy enabled in /etc/default/haproxy.
2018-02-05 16:36:45 DEBUG juju-log Ensuring haproxy enabled in /etc/default/haproxy.
2018-02-05 16:36:45 DEBUG update-status haproxy is running.
2018-02-05 16:36:45 DEBUG update-status * memcached is running
2018-02-05 16:36:45 DEBUG update-status * apache2 is running
...

Any tip what can be producing this continuous restart is appreciated.

Thanks!

José.

[Boroczki, Lajos (boroczki-lajos)]

The following might be the solution:

```
--- hooks/charmhelpers/contrib/hardening/apache/checks/config.py 2017-12-15 10:11:04.162592891 +0000
+++ - 2018-03-23 14:45:11.059832343 +0000
@@ -52,7 +52,7 @@
                                    'mods-available/alias.conf'),
                       context,
                       TEMPLATES_DIR,
- mode=0o0755,
+ mode=0o0640,
                       user='root',
                       service_actions=[{'service': 'apache2',
                                         'actions': ['restart']}]),
```

As line 69 applies a stricter mode to the whole apache_dir directory.

[Ryan Beisner (1chb1n)]

What charm version is this? Can you please provide the git hash from the repo-info file from the running dashboard unit?:
/var/lib/juju/agents/unit-openstack-dashboard-0/charm

Also, what version of juju is this?

I've just completed a simple (single, non-ha, non-ssl) dashboard deploy with harden=apache on trusty-mitka, and it came up cleanly. Granted, that is not a reproduction of the reported model, but it is a data point with a quick baseline.

Thank you.