cinder-volume deployment results in apache2 failure

[Blake Henderson]

OS - Ubuntu 18.04 LTS
Juju charm cinder 276
Openstack Queens

Following instructions for cinder 276 deployment results in cinder-volume in failed state with apache2 down and message "Services not running that should be: apache2"

$ systemctl status apache2
● apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/apache2.service.d
           └─apache2-systemd.conf
   Active: failed (Result: exit-code) since Fri 2018-11-23 18:54:34 UTC; 33min ago
  Process: 11722 ExecStop=/usr/sbin/apachectl stop (code=exited, status=1/FAILURE)
  Process: 21888 ExecReload=/usr/sbin/apachectl graceful (code=exited, status=1/FAILURE)
  Process: 11738 ExecStart=/usr/sbin/apachectl start (code=exited, status=1/FAILURE)
 Main PID: 5440 (code=exited, status=0/SUCCESS)

Nov 23 18:54:34 controller02 systemd[1]: Starting The Apache HTTP Server...
Nov 23 18:54:34 controller02 apachectl[11738]: AH00526: Syntax error on line 6 of /etc/apache2/sites-enabled/wsgi-openstack-api.conf:
Nov 23 18:54:34 controller02 apachectl[11738]: Invalid command 'WSGIDaemonProcess', perhaps misspelled or defined by a module not included in the server configuration
Nov 23 18:54:34 controller02 apachectl[11738]: Action 'start' failed.
Nov 23 18:54:34 controller02 apachectl[11738]: The Apache error log may have more information.
Nov 23 18:54:34 controller02 systemd[1]: apache2.service: Control process exited, code=exited status=1
Nov 23 18:54:34 controller02 systemd[1]: apache2.service: Failed with result 'exit-code'.
Nov 23 18:54:34 controller02 systemd[1]: Failed to start The Apache HTTP Server.

~$ more /etc/apache2/sites-enabled/wsgi-openstack-api.conf
# Configuration file maintained by Juju. Local changes may be overwritten.

Listen 8766
<VirtualHost *:8766>
    WSGIDaemonProcess cinder processes=64 threads=1 user=cinder group=cinder \
                      display-name=%{GROUP}
    WSGIProcessGroup cinder
    WSGIScriptAlias / /usr/bin/cinder-wsgi
    WSGIApplicationGroup %{GLOBAL}
    WSGIPassAuthorization On
    <IfVersion >= 2.4>
      ErrorLogFormat "%{cu}t %M"
    </IfVersion>
    ErrorLog /var/log/apache2/cinder_error.log
    CustomLog /var/log/apache2/cinder_access.log combined

    <Directory /usr/bin>
        <IfVersion >= 2.4>
            Require all granted
        </IfVersion>
        <IfVersion < 2.4>
            Order allow,deny
            Allow from all
        </IfVersion>
    </Directory>
</VirtualHost>

If the lines starting with WSGI are removed, apache2 starts up and cinder-volume functions properly.

I think the juju charm for cinder needs to be modified to not add the WSGI lines.

Test Method
use test1.yaml (below to create a simple openstack queens deployment, then add cinder-api in an lxd and cinder-volume to nodes 0,1,2

Test1.yaml
machines:
  '0':
    series: bionic
    constraints: 'tags=control'
  '1':
    series: bionic
    constraints: 'tags=compute'
  '2':
    series: bionic
    constraints: 'tags=compute'
relations:
- - nova-compute:amqp
  - rabbitmq-server:amqp
- - neutron-gateway:amqp
  - rabbitmq-server:amqp
- - keystone:shared-db
  - mysql:shared-db
- - nova-cloud-controller:identity-service
  - keystone:identity-service
- - glance:identity-service
  - keystone:identity-service
- - neutron-api:identity-service
  - keystone:identity-service
- - neutron-openvswitch:neutron-plugin-api
  - neutron-api:neutron-plugin-api
- - neutron-api:shared-db
  - mysql:shared-db
- - neutron-api:amqp
  - rabbitmq-server:amqp
- - neutron-gateway:neutron-plugin-api
  - neutron-api:neutron-plugin-api
- - glance:shared-db
  - mysql:shared-db
- - glance:amqp
  - rabbitmq-server:amqp
- - nova-cloud-controller:image-service
  - glance:image-service
- - nova-compute:image-service
  - glance:image-service
- - nova-cloud-controller:cloud-compute
  - nova-compute:cloud-compute
- - nova-cloud-controller:amqp
  - rabbitmq-server:amqp
- - nova-cloud-controller:quantum-network-service
  - neutron-gateway:quantum-network-service
- - nova-compute:neutron-plugin
  - neutron-openvswitch:neutron-plugin
- - neutron-openvswitch:amqp
  - rabbitmq-server:amqp
- - openstack-dashboard:identity-service
  - keystone:identity-service
- - nova-cloud-controller:shared-db
  - mysql:shared-db
- - nova-cloud-controller:neutron-api
  - neutron-api:neutron-api
- - ntp:juju-info
  - nova-compute:juju-info
- - ntp:juju-info
  - neutron-gateway:juju-info
series: bionic
services:
  glance:
    annotations:
      gui-x: '250'
      gui-y: '0'
    charm: cs:glance-265
    num_units: 1
    options:
      worker-multiplier: 0.25
      use-syslog: true
    to:
    - lxd:0
  keystone:
    annotations:
      gui-x: '500'
      gui-y: '0'
    charm: cs:keystone-281
    num_units: 1
    options:
      admin-password: openstack
      admin-token: aph9ta3oJesh7Ie4Eiquohpeephah7Iexeer
      worker-multiplier: 0.25
      use-syslog: true
    to:
    - lxd:0
  mysql:
    annotations:
      gui-x: '0'
      gui-y: '250'
    charm: cs:percona-cluster-266
    num_units: 1
    options:
      innodb-buffer-pool-size: 256M
      max-connections: 1000
    to:
    - lxd:0
  neutron-api:
    annotations:
      gui-x: '500'
      gui-y: '500'
    charm: cs:neutron-api-260
    num_units: 1
    options:
      neutron-security-groups: true
      flat-network-providers: physnet1
      worker-multiplier: 0.25
      use-syslog: true
    to:
    - lxd:0
  neutron-gateway:
    annotations:
      gui-x: '0'
      gui-y: '0'
    charm: cs:neutron-gateway-252
    num_units: 1
    options:
      bridge-mappings: physnet1:br-ex
      data-port: br-ex:eth1
      instance-mtu: 1400
      worker-multiplier: 0.25
      use-syslog: true
    to:
    - '0'
  neutron-openvswitch:
    annotations:
      gui-x: '250'
      gui-y: '500'
    charm: cs:neutron-openvswitch-250
    num_units: 0
    options:
     bridge-mappings: physnet1:br-ex
     data-port: br-ex:eth1
     use-syslog: true
  nova-cloud-controller:
    annotations:
      gui-x: '0'
      gui-y: '500'
    charm: cs:nova-cloud-controller-310
    num_units: 1
    options:
      ram-allocation-ratio: 1.5
      console-access-protocol: novnc
      single-nova-consoleauth: false
      network-manager: Neutron
      worker-multiplier: 0.25
      use-syslog: true
    to:
    - lxd:0
  nova-compute:
    annotations:
      gui-x: '250'
      gui-y: '250'
    charm: cs:nova-compute-284
    num_units: 2
    options:
      config-flags: default_ephemeral_format=ext4
      enable-live-migration: true
      enable-resize: true
      migration-auth-type: ssh
      use-syslog: true
    to:
    - '1'
    - '2'
  ntp:
    annotations:
      gui-x: '1000'
      gui-y: '0'
    charm: cs:ntp-24
    num_units: 0
    options:
     source: 10.254.201.15
  openstack-dashboard:
    annotations:
      gui-x: '500'
      gui-y: '-250'
    charm: cs:openstack-dashboard-259
    num_units: 1
    options:
      neutron-network-firewall: true
      neutron-network-lb: false
      ubuntu-theme: 'No'
      default-theme: 'default'
      webroot: /
      use-syslog: true
    to:
    - lxd:0
  rabbitmq-server:
    annotations:
      gui-x: '500'
      gui-y: '250'
    charm: cs:rabbitmq-server-74
    num_units: 1
    options:
     use-syslog: true
    to:
    - lxd:0

When stable: continue.
ysadmin@maasdev:~$ juju status
Model Controller Cloud/Region Version SLA Timestamp
default newcloud-controller rcscloud 2.4.4 unsupported 15:45:04-05:00

App Version Status Scale Charm Store Rev OS Notes
glance 16.0.1 active 1 glance jujucharms 265 ubuntu
keystone 13.0.1 active 1 keystone jujucharms 281 ubuntu
mysql 5.7.20-29.24 active 1 percona-cluster jujucharms 266 ubuntu
neutron-api 12.0.5 active 1 neutron-api jujucharms 260 ubuntu
neutron-gateway 12.0.5 active 1 neutron-gateway jujucharms 252 ubuntu
neutron-openvswitch 12.0.5 active 2 neutron-openvswitch jujucharms 250 ubuntu
nova-cloud-controller 17.0.6 active 1 nova-cloud-controller jujucharms 310 ubuntu
nova-compute 17.0.6 active 2 nova-compute jujucharms 284 ubuntu
ntp 4.2.8p10+dfsg active 3 ntp jujucharms 24 ubuntu
openstack-dashboard 13.0.1 active 1 openstack-dashboard jujucharms 259 ubuntu
rabbitmq-server 3.6.10 active 1 rabbitmq-server jujucharms 74 ubuntu

Unit Workload Agent Machine Public address Ports Message
glance/0* active idle 0/lxd/0 10.254.201.114 9292/tcp Unit is ready
keystone/0* active idle 0/lxd/1 10.254.201.97 5000/tcp Unit is ready
mysql/0* active idle 0/lxd/2 10.254.201.56 3306/tcp Unit is ready
neutron-api/0* active idle 0/lxd/3 10.254.201.87 9696/tcp Unit is ready
neutron-gateway/0* active idle 0 10.254.201.74 Unit is ready
  ntp/1 active idle 10.254.201.74 123/udp Ready
nova-cloud-controller/0* active idle 0/lxd/4 10.254.201.137 8774/tcp,8778/tcp Unit is ready
nova-compute/0* active idle 1 10.254.201.104 Unit is ready
  neutron-openvswitch/0* active idle 10.254.201.104 Unit is ready
  ntp/0* active idle 10.254.201.104 123/udp Ready
nova-compute/1 active idle 2 10.254.201.98 Unit is ready
  neutron-openvswitch/1 active idle 10.254.201.98 Unit is ready
  ntp/2 active idle 10.254.201.98 123/udp Ready
openstack-dashboard/0* active idle 0/lxd/5 10.254.201.69 80/tcp,443/tcp Unit is ready
rabbitmq-server/0* active idle 0/lxd/6 10.254.201.84 5672/tcp Unit is ready

Machine State DNS Inst id Series AZ Message
0 started 10.254.201.74 xtyw3h bionic default Deployed
0/lxd/0 started 10.254.201.114 juju-3d9ceb-0-lxd-0 bionic default Container started
0/lxd/1 started 10.254.201.97 juju-3d9ceb-0-lxd-1 bionic default Container started
0/lxd/2 started 10.254.201.56 juju-3d9ceb-0-lxd-2 bionic default Container started
0/lxd/3 started 10.254.201.87 juju-3d9ceb-0-lxd-3 bionic default Container started
0/lxd/4 started 10.254.201.137 juju-3d9ceb-0-lxd-4 bionic default Container started
0/lxd/5 started 10.254.201.69 juju-3d9ceb-0-lxd-5 bionic default Container started
0/lxd/6 started 10.254.201.84 juju-3d9ceb-0-lxd-6 bionic default Container started
1 started 10.254.201.104 wdmt73 bionic default Deployed
2 started 10.254.201.98 xedarh bionic default Deployed

juju deploy --config=cinder.cfg cinder cinder-api --to lxd:0
juju deploy --config=cinder.cfg cinder cinder-volume --to 0
juju add-relation cinder-api mysql
juju add-relation cinder-api rabbitmq-server
juju add-relation cinder-api keystone
juju status
juju add-relation cinder-api nova-cloud-controller
juju add-relation cinder-volume mysql
juju add-relation cinder-volume rabbitmq-server

Results in:

sysadmin@maasdev:~$ juju status
Model Controller Cloud/Region Version SLA Timestamp
default newcloud-controller rcscloud 2.4.4 unsupported 09:40:49-05:00

App Version Status Scale Charm Store Rev OS Notes
cinder-api 12.0.3 active 1 cinder jujucharms 276 ubuntu
cinder-volume 12.0.3 blocked 1 cinder jujucharms 276 ubuntu
glance 16.0.1 active 1 glance jujucharms 265 ubuntu
keystone 13.0.1 active 1 keystone jujucharms 281 ubuntu
mysql 5.7.20-29.24 active 1 percona-cluster jujucharms 266 ubuntu
neutron-api 12.0.5 active 1 neutron-api jujucharms 260 ubuntu
neutron-gateway 12.0.5 active 1 neutron-gateway jujucharms 252 ubuntu
neutron-openvswitch 12.0.5 active 2 neutron-openvswitch jujucharms 250 ubuntu
nova-cloud-controller 17.0.6 active 1 nova-cloud-controller jujucharms 310 ubuntu
nova-compute 17.0.6 active 2 nova-compute jujucharms 284 ubuntu
ntp 4.2.8p10+dfsg active 3 ntp jujucharms 24 ubuntu
openstack-dashboard 13.0.1 active 1 openstack-dashboard jujucharms 259 ubuntu
rabbitmq-server 3.6.10 active 1 rabbitmq-server jujucharms 74 ubuntu

Unit Workload Agent Machine Public address Ports Message
cinder-api/0* active idle 0/lxd/7 10.254.201.96 8776/tcp Unit is ready
cinder-volume/0* blocked idle 0 10.254.201.84 8776/tcp Services not running that should be: apache2
glance/0* active idle 0/lxd/0 10.254.201.74 9292/tcp Unit is ready
keystone/0* active idle 0/lxd/1 10.254.201.114 5000/tcp Unit is ready
mysql/0* active idle 0/lxd/2 10.254.201.113 3306/tcp Unit is ready
neutron-api/0* active idle 0/lxd/3 10.254.201.69 9696/tcp Unit is ready
neutron-gateway/0* active idle 0 10.254.201.84 Unit is ready
  ntp/1 active idle 10.254.201.84 123/udp Ready
nova-cloud-controller/0* active idle 0/lxd/4 10.254.201.128 8774/tcp,8778/tcp Unit is ready
nova-compute/0* active idle 1 10.254.201.137 Unit is ready
  neutron-openvswitch/0* active idle 10.254.201.137 Unit is ready
  ntp/0* active idle 10.254.201.137 123/udp Ready
nova-compute/1 active idle 2 10.254.201.97 Unit is ready
  neutron-openvswitch/1 active idle 10.254.201.97 Unit is ready
  ntp/2 active idle 10.254.201.97 123/udp Ready
openstack-dashboard/0* active idle 0/lxd/5 10.254.201.98 80/tcp,443/tcp Unit is ready
rabbitmq-server/0* active idle 0/lxd/6 10.254.201.104 5672/tcp Unit is ready

Machine State DNS Inst id Series AZ Message
0 started 10.254.201.84 pakkaa bionic default Deployed
0/lxd/0 started 10.254.201.74 juju-b0229c-0-lxd-0 bionic default Container started
0/lxd/1 started 10.254.201.114 juju-b0229c-0-lxd-1 bionic default Container started
0/lxd/2 started 10.254.201.113 juju-b0229c-0-lxd-2 bionic default Container started
0/lxd/3 started 10.254.201.69 juju-b0229c-0-lxd-3 bionic default Container started
0/lxd/4 started 10.254.201.128 juju-b0229c-0-lxd-4 bionic default Container started
0/lxd/5 started 10.254.201.98 juju-b0229c-0-lxd-5 bionic default Container started
0/lxd/6 started 10.254.201.104 juju-b0229c-0-lxd-6 bionic default Container started
0/lxd/7 started 10.254.201.96 juju-b0229c-0-lxd-7 bionic default Container started
1 started 10.254.201.137 wdmt73 bionic default Deployed
2 started 10.254.201.97 xedarh bionic default Deployed

[Blake Henderson (blakehenderson)]

cut and paste for work method did not come out looking so nice.
Attached file is better.

[Blake Henderson (blakehenderson)]

Also of note:

must manually create the cinder-volumes vg group on system hosting cinder-volume

pvcreate /dev/sdb
vgcreate cinder-volumes /dev/sdb

[Ryan Beisner (1chb1n)]

Can you please provide (sanitized) juju unit logs from cinder-volume/0? The can be found in /var/log/juju/unit*.

[Ryan Beisner (1chb1n)]

Along with this bug, we also need to re-validate and put a scenario test around the following, and update the readme if needed:

https://github.com/openstack/charm-cinder/blob/master/README.md#separate-volume-units-for-scale-out-using-local-storage-and-iscsi

[Blake Henderson (blakehenderson)]

I had tried a different config so will need to re-run the original bundle and cinder install.
Should be able to post new results in an hour or so.

[Blake Henderson (blakehenderson)]

New test run --- same results as before
using same test bundle.

[Blake Henderson (blakehenderson)]

unit-nova-compute-0.log from cinder-volume/0 /var/log/juju/

[Blake Henderson (blakehenderson)]

unit-ntp-0.log - from cinder-volume/0 /var/log/juju ---- cinder volume is on a nova-compute node.

[Blake Henderson (blakehenderson)]

unit-cinder-volume-0.log

[Blake Henderson (blakehenderson)]

unit-neutron-openvswitch-0.log

just to cover all the bases.

[Frode Nordahl (fnordahl)]

Adding a comment to summarize:
When deploying the ``cinder`` charm and making use of the ``enabled-services`` configuration option to split out the cinder services over multiple units it breaks.

There appears to be assumptions in the code that does not take that configuration option into account.

[Andre Ruiz (andre-ruiz)]

This bug is present on a customer installation that requested CinderLVM driver.

The driver itself is working, and although the charm stays at "blocked" state and status is "Services not running that should be: apache2", ignoring that and proceeding with post configuration just works, and volumes can be created and used.

Seems that apache is not really needed in this case. It's only happening on the units with "enabled-services=volume" on compute nodes.

I could help testing the fix if any early release is available.

[Marcelo Subtil Marcal (msmarcal)]

I've hit this bug on a new deployment. I'm subscribing it to field-medium.

[Alex Kavanagh (ajkavanagh)]

Looking at this again, it seems to be to do with LVM storage. As the cinder-lvm charm is now a thing, and LVM support in the cinder charm is deprecated/being removed, is this bug still appropriate?

[Launchpad Janitor (janitor)]

[Expired for OpenStack Cinder Charm because there has been no activity for 60 days.]