fuel-master keystone doesn't start on reboot

Bug #1432874 reported by Andrew Woodward
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
Low
Matthew Mosesohn

Bug Description

[root@nailgun ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  api: "1.0"
  build_number: "179"
  build_id: "2015-03-10_11-56-35"
  nailgun_sha: "cbd28a90c6562e862a8401e9224ef3f1f11ff560"
  python-fuelclient_sha: "871525a39832b7ede62c2c404e0b3f6a994c2f40"
  astute_sha: "2d61ee42ec6dae3181d292c7769d32e40d463893"
  fuellib_sha: "79e43e6f8f7305d44ef2fe36b12772e85ae7a754"
  ostf_sha: "8df5f2fcdae3bc9ea7d700ffd64db820baf51914"
  fuelmain_sha: "9d65f2486fef97e3fee29de03d8b115b03a73aeb"

Steps to reproduce.

1) Install fuel
2) restart master node with hard-reset
3) run `fuel nodes`

expected result:
fuel nodes returns as expected

actual result:

DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient.
Traceback (most recent call last):
  File "/usr/bin/fuel", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/error.py", line 98, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/parser.py", line 226, in main
    parser.parse()
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/parser.py", line 102, in parse
    actions[parsed_params.action].action_func(parsed_params)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/actions/base.py", line 61, in action_func
    method(params)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/actions/node.py", line 285, in list
    node_collection = NodeCollection.get_all()
  File "/usr/lib/python2.6/site-packages/fuelclient/objects/node.py", line 152, in get_all
    return cls(Node.get_all())
  File "/usr/lib/python2.6/site-packages/fuelclient/objects/base.py", line 68, in get_all
    return map(cls.init_with_data, cls.get_all_data())
  File "/usr/lib/python2.6/site-packages/fuelclient/objects/base.py", line 64, in get_all_data
    return cls.connection.get_request(cls.class_api_path)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/error.py", line 98, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 146, in get_request
    headers = {'x-auth-token': self.auth_token}
  File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 57, in auth_token
    if not self.keystone_client.auth_token:
  File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 76, in keystone_client
    self.initialize_keystone_client()
  File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 90, in initialize_keystone_client
    tenant_name=self.tenant)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 147, in __init__
    self.authenticate()
  File "/usr/lib/python2.6/site-packages/keystoneclient/utils.py", line 318, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 425, in authenticate
    resp = self.get_raw_token_from_identity_service(**kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 190, in get_raw_token_from_identity_service
    "%s" % e)
keystoneclient.openstack.common.apiclient.exceptions.AuthorizationFailure: Authorization Failed: Bad Gateway (HTTP 502)
[root@nailgun ~]# fuel node
DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient.
Traceback (most recent call last):
  File "/usr/bin/fuel", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/error.py", line 98, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/parser.py", line 226, in main
    parser.parse()
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/parser.py", line 102, in parse
    actions[parsed_params.action].action_func(parsed_params)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/actions/base.py", line 61, in action_func
    method(params)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/actions/node.py", line 285, in list
    node_collection = NodeCollection.get_all()
  File "/usr/lib/python2.6/site-packages/fuelclient/objects/node.py", line 152, in get_all
    return cls(Node.get_all())
  File "/usr/lib/python2.6/site-packages/fuelclient/objects/base.py", line 68, in get_all
    return map(cls.init_with_data, cls.get_all_data())
  File "/usr/lib/python2.6/site-packages/fuelclient/objects/base.py", line 64, in get_all_data
    return cls.connection.get_request(cls.class_api_path)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/error.py", line 98, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 146, in get_request
    headers = {'x-auth-token': self.auth_token}
  File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 57, in auth_token
    if not self.keystone_client.auth_token:
  File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 76, in keystone_client
    self.initialize_keystone_client()
  File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 90, in initialize_keystone_client
    tenant_name=self.tenant)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 147, in __init__
    self.authenticate()
  File "/usr/lib/python2.6/site-packages/keystoneclient/utils.py", line 318, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 425, in authenticate
    resp = self.get_raw_token_from_identity_service(**kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 190, in get_raw_token_from_identity_service
    "%s" % e)
keystoneclient.openstack.common.apiclient.exceptions.AuthorizationFailure: Authorization Failed: Bad Gateway (HTTP 502)
[root@nailgun ~]# dockerctl status
Invalid selection.
Usage:
  /usr/bin/dockerctl command

Available commands: (Note: work in progress)
  help: show this message
  build: create all Docker containers
  list: list container short names (-l for more output)
  start: start all Docker containers
  restart: restart one or more Docker containers
  stop: stop one or more Docker containers
  shell: start a shell or run a command in a Docker container
  logs: print console log from a container
  revert: reset container to original state
  destroy: destroy one or more containers
  copy: copy files in or out of container
  check: check of container is ready
  backup: back up entire deployment
  restore: restore backed up deployment
[root@nailgun ~]# dockerctl list
nginx
rabbitmq
astute
rsync
keystone
postgres
rsyslog
nailgun
cobbler
ostf
mcollective
[root@nailgun ~]# docker ps | grep key
[root@nailgun ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1fd51ae9c59d fuel/mcollective_6.1:latest "/usr/local/bin/star 2 days ago Up About an hour fuel-core-6.1-mcollective
2c9d5896f106 fuel/cobbler_6.1:latest "/bin/sh -c /usr/loc 2 days ago Up About an hour 53/tcp, 67/tcp, 67/udp, 0.0.0.0:53->53/udp, 0.0.0.0:69->69/udp, 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp fuel-core-6.1-cobbler
92ce1bbf6876 fuel/nginx_6.1:latest "/usr/local/bin/star 2 days ago Up About an hour 0.0.0.0:8000->8000/tcp, 0.0.0.0:8080->8080/tcp fuel-core-6.1-nginx
58f29efcca31 fuel/ostf_6.1:latest "/bin/sh -c /usr/loc 2 days ago Up About an hour 10.110.0.2:8777->8777/tcp, 127.0.0.1:8777->8777/tcp fuel-core-6.1-ostf
3030e5321878 fuel/nailgun_6.1:latest "/bin/sh -c /usr/loc 2 days ago Up About an hour 10.110.0.2:8001->8001/tcp, 127.0.0.1:8001->8001/tcp fuel-core-6.1-nailgun
60d59b4e3f50 fuel/rsyslog_6.1:latest "/usr/local/bin/star 2 days ago Up About an hour 10.110.0.2:514->514/tcp, 127.0.0.1:514->514/tcp, 10.110.0.2:514->514/udp, 127.0.0.1:514->514/udp, 10.110.0.2:25150->25150/tcp, 127.0.0.1:25150->25150/tcp fuel-core-6.1-rsyslog
367f4bc34b20 fuel/astute_6.1:latest "/bin/sh -c /usr/loc 2 days ago Up About an hour fuel-core-6.1-astute
dc0456999aba fuel/rsync_6.1:latest "/bin/sh -c /usr/loc 2 days ago Up About an hour 10.110.0.2:873->873/tcp, 127.0.0.1:873->873/tcp fuel-core-6.1-rsync
0a8ca50cb4e3 fuel/rabbitmq_6.1:latest "/usr/local/bin/star 2 days ago Up About an hour 10.110.0.2:4369->4369/tcp, 127.0.0.1:4369->4369/tcp, 10.110.0.2:5672->5672/tcp, 127.0.0.1:5672->5672/tcp, 127.0.0.1:15672->15672/tcp, 10.110.0.2:15672->15672/tcp, 127.0.0.1:61613->61613/tcp, 10.110.0.2:61613->61613/tcp fuel-core-6.1-rabbitmq
867a06104d27 fuel/postgres_6.1:latest "/usr/local/bin/star 2 days ago Up About an hour 10.110.0.2:5432->5432/tcp, 127.0.0.1:5432->5432/tcp fuel-core-6.1-postgres
[root@nailgun ~]# dockerctl shell keystone
lxc-attach: failed to get the init pid
[root@nailgun ~]# cd /var/log/docker
docker docker-keystone.log docker-ostf.log
docker.1.gz docker-logs/ docker-postgres.log
docker.2.gz docker-mcollective.log docker-rabbitmq.log
docker-astute.log docker-nailgun.log docker-rsync.log
docker-cobbler.log docker-nginx.log docker-rsyslog.log
[root@nailgun ~]# cd /var/log/docker-logs/
astute/ httpd/ naily/ puppet/
audit/ keystone/ nginx/ rabbitmq/
cobbler/ lxc/ ntpstats/ remote/
ConsoleKit/ mcollective.log ostf/ rhsm/
coredump/ mcollective.log.0 ostf.log supervisor/
dnsmasq.log nailgun/ ostf-stdout.log yum.log
[root@nailgun ~]# cd /var/log/docker-logs/
astute/ httpd/ naily/ puppet/
audit/ keystone/ nginx/ rabbitmq/
cobbler/ lxc/ ntpstats/ remote/
ConsoleKit/ mcollective.log ostf/ rhsm/
coredump/ mcollective.log.0 ostf.log supervisor/
dnsmasq.log nailgun/ ostf-stdout.log yum.log
[root@nailgun ~]# cd /var/log/docker-logs/
astute/ httpd/ naily/ puppet/
audit/ keystone/ nginx/ rabbitmq/
cobbler/ lxc/ ntpstats/ remote/
ConsoleKit/ mcollective.log ostf/ rhsm/
coredump/ mcollective.log.0 ostf.log supervisor/
dnsmasq.log nailgun/ ostf-stdout.log yum.log
[root@nailgun ~]# cd /var/log/docker-
docker-astute.log docker-mcollective.log docker-postgres.log
docker-cobbler.log docker-nailgun.log docker-rabbitmq.log
docker-keystone.log docker-nginx.log docker-rsync.log
docker-logs/ docker-ostf.log docker-rsyslog.log
[root@nailgun ~]# cd /var/log/docker-keystone.log
-bash: cd: /var/log/docker-keystone.log: Not a directory
[root@nailgun ~]# less /var/log/docker-keystone.log
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-14T00:16:17Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-14T00:16:27Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-14T00:16:38Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-14T00:16:48Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-14T00:16:58Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-14T00:17:09Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-14T00:17:20Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-14T00:17:30Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-14T00:17:41Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-14T00:17:51Z" level="fatal" msg="cannot enable tty mode on non tty...skipping...
Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T15:48:39Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T15:48:51Z" level="fatal" msg="cannot enable tty mode on non tty input"
time="2015-03-16T21:26:56Z" level="fatal" msg="Get http:///var/run/docker.sock/v1.16/containers/json?all=1: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?"
time="2015-03-16T21:26:56Z" level="fatal" msg="Get http:///var/run/docker.sock/v1.16/containers/json: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?"
time="2015-03-16T21:26:56Z" level="fatal" msg="Post http:///var/run/docker.sock/v1.16/containers/create?name=fuel-core-6.1-keystone: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?"
Get http:///var/run/docker.sock/v1.16/images/fuel-core-6.1-keystone/json: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?Get http:///var/run/docker.sock/v1.16/images/fuel-core-6.1-keystone/json: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T21:26:56Z" level="fatal" msg="Get http:///var/run/docker.sock/v1.16/containers/fuel-core-6.1-keystone/json: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?"
fuel-core-6.1-keystone
Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T21:27:07Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T21:27:18Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T21:27:28Z" level="fatal" msg="cannot enable tty mode on non tty input"
...skipping...
Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T15:48:39Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T15:48:51Z" level="fatal" msg="cannot enable tty mode on non tty input"
time="2015-03-16T21:26:56Z" level="fatal" msg="Get http:///var/run/docker.sock/v1.16/containers/json?all=1: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?"
time="2015-03-16T21:26:56Z" level="fatal" msg="Get http:///var/run/docker.sock/v1.16/containers/json: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?"
time="2015-03-16T21:26:56Z" level="fatal" msg="Post http:///var/run/docker.sock/v1.16/containers/create?name=fuel-core-6.1-keystone: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?"
Get http:///var/run/docker.sock/v1.16/images/fuel-core-6.1-keystone/json: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?Get http:///var/run/docker.sock/v1.16/images/fuel-core-6.1-keystone/json: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T21:26:56Z" level="fatal" msg="Get http:///var/run/docker.sock/v1.16/containers/fuel-core-6.1-keystone/json: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?"
fuel-core-6.1-keystone
Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T21:27:07Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T21:27:18Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T21:27:28Z" level="fatal" msg="cannot enable tty mode on non tty input"
...skipping...
Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T15:48:39Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T15:48:51Z" level="fatal" msg="cannot enable tty mode on non tty input"
time="2015-03-16T21:26:56Z" level="fatal" msg="Get http:///var/run/docker.sock/v1.16/containers/json?all=1: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?"
time="2015-03-16T21:26:56Z" level="fatal" msg="Get http:///var/run/docker.sock/v1.16/containers/json: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?"
time="2015-03-16T21:26:56Z" level="fatal" msg="Post http:///var/run/docker.sock/v1.16/containers/create?name=fuel-core-6.1-keystone: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?"
Get http:///var/run/docker.sock/v1.16/images/fuel-core-6.1-keystone/json: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?Get http:///var/run/docker.sock/v1.16/images/fuel-core-6.1-keystone/json: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T21:26:56Z" level="fatal" msg="Get http:///var/run/docker.sock/v1.16/containers/fuel-core-6.1-keystone/json: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?"
fuel-core-6.1-keystone
Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T21:27:07Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T21:27:18Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T21:27:28Z" level="fatal" msg="cannot enable tty mode on non tty input"
 ESCOD
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T15:48:39Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T15:48:51Z" level="fatal" msg="cannot enable tty mode on non tty input"
time="2015-03-16T21:26:56Z" level="fatal" msg="Get http:///var/run/docker.sock/v1.16/containers/json?all=1: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?"
time="2015-03-16T21:26:56Z" level="fatal" msg="Get http:///var/run/docker.sock/v1.16/containers/json: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?"
time="2015-03-16T21:26:56Z" level="fatal" msg="Post http:///var/run/docker.sock/v1.16/containers/create?name=fuel-core-6.1-keystone: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?"
Get http:///var/run/docker.sock/v1.16/images/fuel-core-6.1-keystone/json: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?Get http:///var/run/docker.sock/v1.16/images/fuel-core-6.1-keystone/json: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T21:26:56Z" level="fatal" msg="Get http:///var/run/docker.sock/v1.16/containers/fuel-core-6.1-keystone/json: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?"
fuel-core-6.1-keystone
Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T21:27:07Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T21:27:18Z" level="fatal" msg="cannot enable tty mode on non tty input"
fuel-core-6.1-keystone is already running.
Attaching to container fuel-core-6.1-keystone...
time="2015-03-16T21:27:28Z" level="fatal" msg="cannot enable tty mode on non tty input"
[root@nailgun ~]# dockerctl start
fuel-core-6.1-postgres is already running.
checking container postgres
checking with command "! shell_container postgres pgrep puppet"
checking with command "PGPASSWORD=G01wQatu shell_container postgres psql -h 10.110.0.2 -U "nailgun" "nailgun" -c '\copyright' 2>&1 1>/dev/null"
postgres is ready.
fuel-core-6.1-rabbitmq is already running.
checking container rabbitmq
checking with command "! shell_container rabbitmq pgrep puppet"
checking with command "curl -f -L -i -u "naily:l4unTtqY" http://10.110.0.2:15672/api/nodes 1>/dev/null 2>&1"
checking with command "curl -f -L -u "mcollective:0P8Dr2Ui" -s http://10.110.0.2:15672/api/exchanges | grep -qw 'mcollective_broadcast'"
checking with command "curl -f -L -u "mcollective:0P8Dr2Ui" -s http://10.110.0.2:15672/api/exchanges | grep -qw 'mcollective_directed'"
rabbitmq is ready.
fuel-core-6.1-keystone
checking container keystone
checking with command "! shell_container keystone pgrep puppet"
checking with command "shell_container keystone keystone --os-auth-url "http://10.110.0.2:35357/v2.0" --os-username "nailgun" --os-password "7qBusRS5" token-get &>/dev/null"
try number 1
return code is 1
try number 2
return code is 1
try number 3
return code is 1
try number 4
return code is 1
try number 5
return code is 1
try number 6
return code is 1
keystone is ready.
fuel-core-6.1-rsync is already running.
checking container rsync
checking with command "! shell_container rsync pgrep puppet"
checking with command "shell_container rsync netstat -ntl | grep -q 873"
rsync is ready.
fuel-core-6.1-astute is already running.
checking container astute
checking with command "! shell_container astute pgrep puppet"
checking with command "shell_container astute ps aux | grep -q 'astuted'"
checking with command "curl -f -L -u "naily:l4unTtqY" -s http://10.110.0.2:15672/api/exchanges | grep -qw 'nailgun'"
checking with command "curl -f -L -u "naily:l4unTtqY" -s http://10.110.0.2:15672/api/exchanges | grep -qw 'naily_service'"
astute is ready.
fuel-core-6.1-rsyslog is already running.
checking container rsyslog
checking with command "! shell_container rsyslog pgrep puppet"
checking with command "shell_container rsyslog netstat -nl | grep -q 514"
rsyslog is ready.
fuel-core-6.1-nailgun is already running.
checking container nailgun
checking with command "! shell_container nailgun pgrep puppet"
checking with command "shell_container nailgun supervisorctl status nailgun | grep -q RUNNING"
nailgun is ready.
fuel-core-6.1-ostf is already running.
checking container ostf
checking with command "! shell_container ostf pgrep puppet"
checking with command "egrep -q ^[2-4][0-9]? < <(curl --connect-timeout 1 -s -w '%{http_code}' http://10.110.0.2:8777/ostf/not_found -o /dev/null)"
ostf is ready.
fuel-core-6.1-nginx is already running.
checking container nginx
checking with command "! shell_container nginx pgrep puppet"
checking with command "shell_container nginx ps aux | grep -q nginx"
nginx is ready.
fuel-core-6.1-cobbler is already running.
Shutting down dhcrelay: [ OK ]
Starting dhcrelay: [ OK ]
checking container cobbler
checking with command "! shell_container cobbler pgrep puppet"
checking with command "shell_container cobbler ps aux | grep -q 'cobblerd -F'"
checking with command "shell_container cobbler cobbler profile find --name=centos* | grep -q centos && shell_container cobbler cobbler profile find --name=ubuntu* | grep -q ubuntu && shell_container cobbler cobbler profile find --name=bootstrap* | grep -q bootstrap"
cobbler is ready.
fuel-core-6.1-mcollective is already running.
checking container mcollective
checking with command "! shell_container mcollective pgrep puppet"
checking with command "shell_container mcollective ps aux | grep -q mcollectived"
mcollective is ready.

Revision history for this message
Andrew Woodward (xarses) wrote :
Ryan Moe (rmoe)
Changed in fuel:
status: New → Confirmed
Revision history for this message
Ryan Moe (rmoe) wrote :

I think the key error from the logs (supervisord.log) is this:

2015-03-16 23:13:28,374 DEBG 'docker-keystone' stdout output:
time="2015-03-16T23:13:28Z" level="fatal" msg="cannot enable tty mode on non tty input"

This happens for all of the containers. supervisorctl status shows the containers constantly flapping between RUNNING and STARTING states.

This is happening on my environment deployed from ISO 191 and the master node has not been rebooted.

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Matthew Mosesohn (raytrac3r)
Revision history for this message
Matthew Mosesohn (raytrac3r) wrote :

Can you reproduce? Docker attach is now fixed after https://review.openstack.org/#/c/121559/

Changed in fuel:
status: Confirmed → Incomplete
Revision history for this message
Matthew Mosesohn (raytrac3r) wrote :

Fuel CLI is uanvailable for the first ~4 minutes, and then it worked after hard reset.
[root@fuel ~]# fuel nodes
DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient.

id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|--------|------|---------|----|-----|-------|---------------|--------|---------

[root@fuel ~]#
[root@fuel ~]# uptime
 15:14:27 up 4 min, 1 user, load average: 8.30, 5.45, 2.32

Changed in fuel:
status: Incomplete → Invalid
importance: High → Low
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.