Master node deployment failed on building keystone container

[Andrey Sledzinskiy]

Steps:
1. Start installing fuel

Actual result - fuel deployment failed on building keystone container:
[mNotice: /Stage[main]/Docker/Exec[build docker containers]/returns: ERROR: keystone failed to start.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "293"
  build_id: "293"
  nailgun_sha: "16a39d40120dd4257698795f12de4ae8200b1778"
  python-fuelclient_sha: "2864459e27b0510a0f7aedac6cdf27901ef5c481"
  fuel-agent_sha: "082a47bf014002e515001be05f99040437281a2d"
  fuel-nailgun-agent_sha: "d7027952870a35db8dc52f185bb1158cdd3d1ebd"
  astute_sha: "a717657232721a7fafc67ff5e1c696c9dbeb0b95"
  fuel-library_sha: "0c5a39a43bffaddc91b46179d16e1bb2dd85bc0c"
  fuel-ostf_sha: "1f08e6e71021179b9881a824d9c999957fcc7045"
  fuelmain_sha: "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"

[Matthew Mosesohn (raytrac3r)]

The logs don't contain why keystone failed. We should include /var/lib/docker/containers/ (only contains ~1-10mb of metadata and logs) in the snapshot for this test. Can we reproduce this env and get the output of `dockerctl logs keystone`?

[Matthew Mosesohn (raytrac3r)]

It looks like patching testing scripts are trying to write to yum before we can proactively run rpm --rebuilddb in start.sh.

Another app is currently holding the yum lock; waiting for it to exit...
  The other application is: yum
    Memory : 17 M RSS (211 MB VSZ)
    Started: Mon Sep 14 08:55:49 2015 - 1:31:23 ago
    State : Running, pid: 575
DEBU[0000] [stream] End of stdout

[root@nailgun ~]# dockerctl shell keystone
[root@nailgun ~]# ps waux | grep yum
root 574 0.0 0.0 11300 1176 ? S+ 08:55 0:00 sh -c /usr/bin/yum list | grep fuel-misc
root 575 0.1 0.8 216396 17568 ? S+ 08:55 0:06 /usr/bin/python /usr/bin/yum list
root 2343 0.0 0.0 6380 688 ? S+ 10:27 0:00 grep yum
[root@nailgun ~]# ps waux | grep puppet
root 23 0.0 3.3 316464 68492 ? Sl+ 08:55 0:02 /usr/bin/ruby /usr/bin/puppet apply --detailed-exitcodes -v /etc/puppet/modules/nailgun/examples/keystone-only.pp
root 2345 0.0 0.0 6380 684 ? S+ 10:27 0:00 grep puppet
[root@nailgun ~]# ls
[root@nailgun ~]# yum info
/var/lib/rpm/Packages: unexpected file type or format
error: cannot open Packages index using db3 - Invalid argument (22)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:
Error: rpmdb open failed
--------------
But it seems there are some issues on the actual VM for Fuel Master itself:
[root@nailgun ~]# dockerctl destroy keystone
Stopping keystone...
fuel-core-7.0-keystone
Error response from daemon: Cannot destroy container fuel-core-7.0-keystone: Driver devicemapper failed to remove root filesystem 01b3d040dcd6eef87611548846e0c3ec53caa8c8520807d2479470d95a896784: Error running DeleteDevice dm_task_run failed

Let's try to redeploy and see if we can reproduce this again.

[Matthew Mosesohn (raytrac3r)]

This didn't reproduce when rerunning BVT. It looks like we should add a release notes item and suggest to users who encounter this bug
The release note item should read:
Sometimes Fuel Master node deployment fails due to a hanging yum process inside a Docker container. Sometimes this can be caused by a corrupted Docker storage backend. If the container itself cannot be deleted via `dockerctl delete $CONTAINERNAME` due to an error, such as "Error response from daemon: Cannot destroy container fuel-core-7.0-keystone: Driver devicemapper failed to remove root filesystem 01b3d040dcd6eef87611548846e0c3ec53caa8c8520807d2479470d95a896784: Error running DeleteDevice dm_task_run failed", you should reinstall Fuel Master and deploy again.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/223462

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/223462
Committed: https://git.openstack.org/cgit/stackforge/fuel-docs/commit/?id=b0a61f6ec03beb0a2084986f5fda78d6f8c0875b
Submitter: Jenkins
Branch: master

commit b0a61f6ec03beb0a2084986f5fda78d6f8c0875b
Author: evkonstantinov <email address hidden>
Date: Tue Sep 15 10:43:46 2015 +0300

    Add Fuel Master node deployment issue to relnotes

    Change-Id: I41da04d725575f3ec3badae50b0587fd39161ae9
    Related-Bug:#1495403

[Matthew Mosesohn (raytrac3r)]

This bug is related to docker not releasing ports sometimes. It's quite a rare, random bug, and corrects itself within 7 minutes. Maybe CentOS 7 with Docker 1.7.1 will resolve this. Leaving incomplete for now.

[Matthew Mosesohn (raytrac3r)]

Moving to incomplete because we don't have 7.0 to 8.0 to test upgrade yet

[Matthew Mosesohn (raytrac3r)]

Marked as invalid because 1 month without reproducing