master node setup failed with docker containers in status Not created

[Aleksandra Fedorova]

'setup master' test group failed on latest 7.0 iso. After reverting failed snapshot you can login to master node via SSH and see that

  docker load -i /var/www/nailgun/docker/images/fuel-images.tar

is running. Then it ends and '/bin/bash /usr/bin/dockerctl check' process starts which seems to take forever.

# cat /etc/nailgun/version.yaml
VERSION:
  feature_groups:
    - experimental
  production: "docker"
  release: "7.0"
  openstack_version: "2014.2.2-7.0"
  api: "1.0"
  build_number: "9"
  build_id: "2015-06-20_10-22-51"
  nailgun_sha: "81cc5db978eff19271512266ce23143b229ef4d8"
  python-fuelclient_sha: "8e221ab4b1565fc03382debfff1096768170362c"
  astute_sha: "776157f722b13aff5f59bc098cf948793e6498ef"
  fuel-library_sha: "7e10ed56bc66a8b4a180ff763d6512e6dcc0ba41"
  fuel-ostf_sha: "bac04dccb22a1b247a607b618ccc96c6fcdef2c8"
  fuelmain_sha: "656f39cca441e1593b7ea536ce7a35e10cf48884"

[root@nailgun ~]# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
fuel/postgres_7.0 latest ae4e5d4deab7 2 hours ago 935.1 MB
fuel/rabbitmq_7.0 latest 7f042336fe96 2 hours ago 940.3 MB
fuel/rsyslog_7.0 latest 695b035ef9d1 2 hours ago 937.6 MB
fuel/rsync_7.0 latest 97c379be1568 2 hours ago 937.5 MB
fuel/ostf_7.0 latest 031f191739e4 2 hours ago 1.016 GB
fuel/nginx_7.0 latest 8b0eee7731c9 2 hours ago 937.5 MB
fuel/keystone_7.0 latest 4cc5b3e80630 2 hours ago 937.4 MB
fuel/nailgun_7.0 latest dc8a170803c1 2 hours ago 991.9 MB
fuel/mcollective_7.0 latest 72db7468d98d 2 hours ago 1.08 GB
fuel/cobbler_7.0 latest e32d6425f4ee 2 hours ago 975.1 MB
fuel/astute_7.0 latest 2b8351c4a589 2 hours ago 1.002 GB
fuel/centos latest 6a1d11699597 2 hours ago 924.7 MB
busybox buildroot-2014.02 8c2e06607696 9 weeks ago 2.433 MB
busybox latest 2d8e5b282c81 14 months ago 2.489 MB

[root@nailgun ~]# dockerctl list -l
Name Image Status Full container name
nginx fuel/nginx_7.0 Not created fuel-core-7.0-nginx
rabbitmq fuel/rabbitmq_7.0 Not created fuel-core-7.0-rabbitmq
astute fuel/astute_7.0 Not created fuel-core-7.0-astute
rsync fuel/rsync_7.0 Not created fuel-core-7.0-rsync
keystone fuel/keystone_7.0 Not created fuel-core-7.0-keystone
postgres fuel/postgres_7.0 Not created fuel-core-7.0-postgres
rsyslog fuel/rsyslog_7.0 Not created fuel-core-7.0-rsyslog
nailgun fuel/nailgun_7.0 Not created fuel-core-7.0-nailgun
cobbler fuel/cobbler_7.0 Not created fuel-core-7.0-cobbler
ostf fuel/ostf_7.0 Not created fuel-core-7.0-ostf
mcollective fuel/mcollective_7.0 Not created fuel-core-7.0-mcollective

From /var/log/docker

time="2015-06-20T13:00:38Z" level="error" msg="Handler for GET /containers/{name:.*}/json returned error: No such container: fuel-core-7.0-postgres"
time="2015-06-20T13:00:38Z" level="error" msg="HTTP Error: statusCode=404 No such container: fuel-core-7.0-postgres"
No such image: fuel-core-7.0-postgres
time="2015-06-20T13:00:38Z" level="error" msg="Handler for GET /images/{name:.*}/json returned error: No such image: fuel-core-7.0-postgres"
time="2015-06-20T13:00:38Z" level="error" msg="HTTP Error: statusCode=404 No such image: fuel-core-7.0-postgres"
No such container: postgres
time="2015-06-20T13:00:38Z" level="error" msg="Handler for GET /containers/{name:.*}/json returned error: No such container: postgres"
time="2015-06-20T13:00:38Z" level="error" msg="HTTP Error: statusCode=404 No such container: postgres"
No such image: postgres
time="2015-06-20T13:00:38Z" level="error" msg="Handler for GET /images/{name:.*}/json returned error: No such image: postgres"
time="2015-06-20T13:00:38Z" level="error" msg="HTTP Error: statusCode=404 No such image: postgres"

Same failures appear with community builds BVT (both Ubuntu and CentOS)
https://ci.fuel-infra.org/job/fuel_community.centos.bvt_1/192/console

[Bartłomiej Piotrowski (bpiotrowski)]

I can't reproduce it on 7.0-10 and 7.0-11. Could it be caused by https://bugs.launchpad.net/fuel/+bug/1466640 ?

[Aleksandra Fedorova (bookwar)]

With @mattymo's help we found that the issue is:

  pkg_resources.DistributionNotFound: ethtool>=0.6

Bug was introduced by the latest commit in fuel-main, so it didn't appear on 7.0-11, only 7.0-12.

[Aleksandra Fedorova (bookwar)]

https://review.openstack.org/#/c/193810/

[Aleksandra Fedorova (bookwar)]

ISO with patch above passes both BVT's

OSCI team, please check why test-requirements job is failing and merge the revert