An OpenPower node within an Ubuntu Openstack HA cluster has it's base ext4 filesystem corrupted

[Jason Furmanek]

We have a Ubuntu Openstack HA cluster running on 5 OpenPower nodes (POWER8).

This is on 14.04 with Openstack Liberty

It was all working just peachy, when node 5 starting having issues and became inaccessible.
It seems the file system on the local disk (ext4) has some errors and requires a fsck/e2fsck.
The system does booot, however, the erros in the filesystem seems to have affected the SSH keys so I cannot get into the system. The juju agent does not start.

This is related to bug 1578471.

This bug is to determine if the corrupt filesystem was just from a failing disk, or if the software is somehow in fault.
As this is an Openstack HA environment, this node is running containers with:
Cinder
glance
horizon
neutron
rabbitmq

ceph-osd and nova are running on the host OS itself.

I've not attempted to "fix" this node yet, by running fsck somehow, or just doing a redeploy. I can do either of those, if it's recommended.

bundle.yaml
==========

machines:
  '0':
    constraints: arch=ppc64el
    series: trusty
  '1':
    constraints: { arch=ppc64el, tags=compute }
    series: trusty
  '2':
    constraints: { arch=ppc64el, tags=compute }
    series: trusty
  '3':
    constraints: { arch=ppc64el, tags=compute }
    series: trusty
  '4':
    constraints: { arch=ppc64el, tags=compute }
    series: trusty
  '5':
    constraints: { arch=ppc64el, tags=compute }
    series: trusty
  '6':
    constraints: { arch=ppc64el, tags=neutron-gateway }
    series: trusty

relations:
- - nova-compute:amqp
  - rabbitmq-server:amqp
- - neutron-gateway:amqp
  - rabbitmq-server:amqp
- - keystone:shared-db
  - mysql:shared-db
- - nova-cloud-controller:identity-service
  - keystone:identity-service
- - glance:identity-service
  - keystone:identity-service
- - neutron-api:identity-service
  - keystone:identity-service
- - neutron-openvswitch:neutron-plugin-api
  - neutron-api:neutron-plugin-api
- - neutron-api:shared-db
  - mysql:shared-db
- - neutron-api:amqp
  - rabbitmq-server:amqp
- - neutron-gateway:neutron-plugin-api
  - neutron-api:neutron-plugin-api
- - glance:shared-db
  - mysql:shared-db
- - glance:amqp
  - rabbitmq-server:amqp
- - nova-cloud-controller:image-service
  - glance:image-service
- - nova-compute:image-service
  - glance:image-service
- - nova-cloud-controller:cloud-compute
  - nova-compute:cloud-compute
- - nova-cloud-controller:amqp
  - rabbitmq-server:amqp
- - nova-cloud-controller:quantum-network-service
  - neutron-gateway:quantum-network-service
- - nova-compute:neutron-plugin
  - neutron-openvswitch:neutron-plugin
- - neutron-openvswitch:amqp
  - rabbitmq-server:amqp
- - openstack-dashboard:identity-service
  - keystone:identity-service
- - nova-cloud-controller:shared-db
  - mysql:shared-db
- - nova-cloud-controller:neutron-api
  - neutron-api:neutron-api
- - cinder:image-service
  - glance:image-service
- - cinder:amqp
  - rabbitmq-server:amqp
- - cinder:identity-service
  - keystone:identity-service
- - cinder:cinder-volume-service
  - nova-cloud-controller:cinder-volume-service
- - cinder-ceph:storage-backend
  - cinder:storage-backend
- - ceph:client
  - nova-compute:ceph
- - cinder:shared-db
  - mysql:shared-db
- - ceph:client
  - cinder-ceph:ceph
- - ceph:client
  - glance:ceph
- - ceph-osd:mon
  - ceph:osd
- - ntp:juju-info
  - nova-compute:juju-info
- - ntp:juju-info
  - neutron-gateway:juju-info
- - ceilometer:amqp
  - rabbitmq-server:amqp
- - ceilometer-agent:ceilometer-service
  - ceilometer:ceilometer-service
- - ceilometer:identity-service
  - keystone:identity-service
- - ceilometer:identity-notifications
  - keystone:identity-notifications
- - ceilometer-agent:nova-ceilometer
  - nova-compute:nova-ceilometer
- - ceilometer:shared-db
  - mongodb:database
- - memcached
  - nova-cloud-controller
- - hacluster-ceilometer
  - ceilometer
- - hacluster-cinder
  - cinder
- - hacluster-glance
  - glance
- - hacluster-keystone
  - keystone
#- - hacluster-mysql
# - mysql
- - hacluster-neutron-api
  - neutron-api
- - hacluster-nova-cloud-controller
  - nova-cloud-controller
- - hacluster-openstack-dashboard
  - openstack-dashboard

series: trusty
services:
  ceilometer:
    annotations:
      gui-x: '1288.8744298356794'
      gui-y: '0.7040786325134718'
    charm: cs:trusty/ceilometer
    num_units: 3
    options:
      vip: 10.0.84.19
      openstack-origin: cloud:trusty-liberty
    to:
    - lxc:2
    - lxc:3
    - lxc:4
  ceilometer-agent:
    annotations:
      gui-x: '1288.9999389648438'
      gui-y: '503'
    charm: cs:trusty/ceilometer-agent
  ceph:
    annotations:
      gui-x: '750'
      gui-y: '500'
    charm: cs:trusty/ceph
    num_units: 3
    options:
      fsid: cbd8508e-d726-4785-bff9-2fbf4af2df61
      monitor-secret: AQDxeg9XUMRsKRAAyfClczi5hEV/3j0CuIN8dA==
      osd-devices: '/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:0:0 /dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:1:0 /dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:2:0 /dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:3:0 /dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:4:0 /dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:5:0 /dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:6:0 /dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:7:0'
      osd-reformat: 'yes'
      source: cloud:trusty-liberty
    to:
    - '1'
    - '2'
    - '3'
  ceph-osd:
    annotations:
      gui-x: '1000'
      gui-y: '500'
    charm: cs:trusty/ceph-osd
    num_units: 2
    options:
      osd-devices: '/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:0:0 /dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:1:0 /dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:2:0 /dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:3:0 /dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:4:0 /dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:5:0 /dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:6:0 /dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:7:0'
      osd-reformat: 'yes'
      source: cloud:trusty-liberty
    to:
    - '4'
    - '5'
  cinder:
    annotations:
      gui-x: '750'
      gui-y: '0'
    charm: cs:trusty/cinder
    num_units: 3
    options:
      vip: 10.0.84.15
      block-device: None
      glance-api-version: 2
      openstack-origin: cloud:trusty-liberty
    to:
    - lxc:4
    - lxc:5
    - lxc:1
  cinder-ceph:
    annotations:
      gui-x: '750'
      gui-y: '250'
    charm: cs:trusty/cinder-ceph
    num_units: 0
  glance:
    annotations:
      gui-x: '250'
      gui-y: '0'
    charm: cs:trusty/glance
    num_units: 3
    options:
      vip: 10.0.84.14
      openstack-origin: cloud:trusty-liberty
    to:
    - lxc:5
    - lxc:1
    - lxc:2
  keystone:
    annotations:
      gui-x: '500'
      gui-y: '0'
    charm: cs:trusty/keystone
    num_units: 3
    options:
      vip: 10.0.84.13
      admin-password: openstack
      openstack-origin: cloud:trusty-liberty
    to:
    - lxc:1
    - lxc:2
    - lxc:3
  mongodb:
    annotations:
      gui-x: '1287.9999389648438'
      gui-y: '251.24996948242188'
    charm: cs:trusty/mongodb
    num_units: 3
    to:
    - lxc:2
    - lxc:3
    - lxc:4
  mysql:
    annotations:
      gui-x: '0'
      gui-y: '250'
    #charm: cs:trusty/percona-cluster
    charm: cs:trusty/mysql
    num_units: 1
    options:
      #vip: 10.0.84.12
      max-connections: 20000
      #innodb-buffer-pool-size: 4G
      dataset-size: 4G
      #root-password: NHYEGOYnHg9qJ2ByutPrI5P5UmQQklaG
      #sst-password: c78alhN2tG6hpQ46dh1T6rHV19nax86M
      #min-cluster-size: 3
      #lp1366997-workaround: true
    to:
    - lxc:1
  neutron-api:
    annotations:
      gui-x: '500'
      gui-y: '500'
    charm: cs:trusty/neutron-api
    num_units: 3
    options:
      vip: 10.0.84.17
      neutron-security-groups: true
      openstack-origin: cloud:trusty-liberty
      dhcp-agents-per-network: 2
    to:
    - lxc:4
    - lxc:5
    - lxc:1
  neutron-gateway:
    annotations:
      gui-x: '500'
      gui-y: '500'
    charm: cs:trusty/neutron-gateway
    num_units: 1
    options:
      ext-port: eth1
      openstack-origin: cloud:trusty-liberty
      instance-mtu: 1400
    to:
    - 6
  neutron-openvswitch:
    annotations:
      gui-x: '250'
      gui-y: '500'
    charm: cs:trusty/neutron-openvswitch
    num_units: 0
    options:
      enable-local-dhcp-and-metadata: true
  nova-cloud-controller:
    annotations:
      gui-x: '0'
      gui-y: '500'
    charm: cs:trusty/nova-cloud-controller
    num_units: 3
    options:
      vip: 10.0.84.16
      network-manager: Neutron
      openstack-origin: cloud:trusty-liberty
# console-access-protocol: novnc
      console-access-protocol: None
      ram-allocation-ratio: 1.0
      single-nova-consoleauth: false
    to:
    - lxc:5
    - lxc:1
    - lxc:2
  memcached:
    charm: cs:trusty/memcached
    num_units: 3
    options:
      allow-ufw-ip6-softfail: true
    to:
    - lxc:1
    - lxc:2
    - lxc:3
  nova-compute:
    annotations:
      gui-x: '250'
      gui-y: '250'
    charm: cs:trusty/nova-compute
    num_units: 5
    options:
      enable-live-migration: true
      enable-resize: true
      manage-neutron-plugin-legacy-mode: false
      migration-auth-type: ssh
      openstack-origin: cloud:trusty-liberty
    to:
    - '1'
    - '2'
    - '3'
    - '4'
    - '5'
  ntp:
    annotations:
      gui-x: '1000'
      gui-y: '0'
    charm: cs:trusty/ntp
    num_units: 0
  openstack-dashboard:
    annotations:
      gui-x: '500'
      gui-y: '-250'
    charm: cs:trusty/openstack-dashboard
    num_units: 3
    options:
      vip: 10.0.84.11
      openstack-origin: cloud:trusty-liberty
      webroot: /
      secret: OPWEY1P8wN029oTH4ZhWGy5JCcnMBn0I
    to:
    - lxc:2
    - lxc:3
    - lxc:4
  rabbitmq-server:
    annotations:
      gui-x: '500'
      gui-y: '250'
    charm: cs:trusty/rabbitmq-server
    num_units: 3
    options:
      source: cloud:trusty-liberty
      cluster-partition-handling: pause_minority
      min-cluster-size: 3
    to:
    - lxc:3
    - lxc:4
    - lxc:5
  hacluster-ceilometer:
    charm: cs:trusty/hacluster
    num_units: 0
    options:
      corosync_key: '1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
      cluster_count: 3
      corosync_transport: unicast
  hacluster-cinder:
    charm: cs:trusty/hacluster
    num_units: 0
    options:
      corosync_key: '1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
      cluster_count: 3
      corosync_transport: unicast
  hacluster-glance:
    charm: cs:trusty/hacluster
    num_units: 0
    options:
      corosync_key: '1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
      cluster_count: 3
      corosync_transport: unicast
  hacluster-keystone:
    charm: cs:trusty/hacluster
    num_units: 0
    options:
      corosync_key: '1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
      cluster_count: 3
      corosync_transport: unicast
# hacluster-mysql:
# charm: cs:trusty/hacluster
# num_units: 0
# options:
# corosync_key: '1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
# cluster_count: 3
# corosync_transport: unicast
  hacluster-neutron-api:
    charm: cs:trusty/hacluster
    num_units: 0
    options:
      corosync_key: '1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
      cluster_count: 3
      corosync_transport: unicast
  hacluster-nova-cloud-controller:
    charm: cs:trusty/hacluster
    num_units: 0
    options:
      corosync_key: '1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
      cluster_count: 3
      corosync_transport: unicast
  hacluster-openstack-dashboard:
    charm: cs:trusty/hacluster
    num_units: 0
    options:
      corosync_key: '1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
      cluster_count: 3
      corosync_transport: unicast

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1578475/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.