juju deploy failure WebSocketConnection closed

[Larry Michel]

This is error during openstack deployment:

+ . ./pipeline_parameters
++ export OPENSTACK_RELEASE=icehouse
++ OPENSTACK_RELEASE=icehouse
++ export COMPUTE=nova-lxc
++ COMPUTE=nova-lxc
++ export BLOCK_STORAGE=cinder-iscsi
++ BLOCK_STORAGE=cinder-iscsi
++ export IMAGE_STORAGE=glance-ceph
++ IMAGE_STORAGE=glance-ceph
++ export PIPELINE_ID=03494fef-97ee-458b-83a4-bad35e3e3844
++ PIPELINE_ID=03494fef-97ee-458b-83a4-bad35e3e3844
++ export NETWORKING=neutron-nvp
++ NETWORKING=neutron-nvp
++ export UBUNTU_RELEASE=trusty
++ UBUNTU_RELEASE=trusty

Bootstrap succeeded:

2015-02-03 17:37:12,530 [INFO] oil_ci.juju.client: Boostrapping new environment
Bootstrapping environment "ci-oil-slave10"
Starting new instance for initial state server
Launching instance
WARNING no architecture was specified, acquiring an arbitrary node
WARNING no architecture was specified, acquiring an arbitrary node
WARNING no architecture was specified, acquiring an arbitrary node
 - /MAAS/api/1.0/nodes/node-a2ed9f80-c4cd-11e3-8102-00163efc5068/
Installing Juju agent on bootstrap instance
Waiting for address
Attempting to connect to hayward-11.oil:22
Attempting to connect to hayward-11.oil:22
Attempting to connect to 10.245.0.161:22
Warning: Permanently added '10.245.0.161' (ECDSA) to the list of known hosts.
Logging to /var/log/cloud-init-output.log on remote host
Installing add-apt-repository
Adding apt repository: deb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/cloud-tools main
Running apt-get update
Running apt-get upgrade
Installing package: curl
Installing package: cpu-checker
Installing package: bridge-utils
Installing package: rsyslog-gnutls
Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz <[https://streams.canonical.com/juju/tools/releases/juju-1.21.1-precise-amd64.tgz]>
Bootstrapping Juju machine agent
Starting Juju machine agent (jujud-machine-0)
Bootstrap complete

And the failure:

2015-02-03 17:49:56,354 [DEBUG] deployer.import: Getting charms...
2015-02-03 17:49:56,355 [DEBUG] deployer.charm: Branching charm lp:charms/mongodb @ trusty/mongodb
2015-02-03 17:49:58,839 [DEBUG] deployer.charm: Branching charm lp:~openstack-charmers/charms/trusty/nova-cloud-controller/next @ trusty/nova-cloud-controller
2015-02-03 17:50:01,271 [DEBUG] deployer.charm: Branching charm lp:~lutostag/charms/trusty/ceph/fix-six-bug @ trusty/ceph
2015-02-03 17:50:10,808 [DEBUG] deployer.charm: Branching charm lp:~openstack-charmers/charms/trusty/heat/next @ trusty/heat
2015-02-03 17:50:17,109 [DEBUG] deployer.charm: Branching charm lp:charms/neutron-openvswitch @ trusty/neutron-openvswitch
2015-02-03 17:50:19,876 [DEBUG] deployer.charm: Branching charm lp:~lutostag/charms/trusty/mysql/fix-six-bug @ trusty/mysql
2015-02-03 17:50:23,105 [DEBUG] deployer.charm: Branching charm lp:charms/neutron-api @ trusty/neutron-api
2015-02-03 17:50:26,148 [DEBUG] deployer.charm: Branching charm lp:~openstack-charmers/charms/trusty/glance/next @ trusty/glance
2015-02-03 17:50:30,087 [DEBUG] deployer.charm: Branching charm lp:charms/ceilometer @ trusty/ceilometer
2015-02-03 17:50:33,287 [DEBUG] deployer.charm: Branching charm lp:~lutostag/charms/trusty/nova-compute/oil @ trusty/nova-compute
2015-02-03 17:50:36,200 [DEBUG] deployer.charm: Branching charm lp:charms/ceilometer-agent @ trusty/ceilometer-agent
2015-02-03 17:50:39,239 [DEBUG] deployer.charm: Branching charm lp:charms/rabbitmq-server @ trusty/rabbitmq-server
2015-02-03 17:50:41,770 [DEBUG] deployer.charm: Branching charm lp:~gnuoy/charms/trusty/keystone/next-1385105 @ trusty/keystone
2015-02-03 17:50:44,320 [DEBUG] deployer.charm: Branching charm lp:charms/openstack-dashboard @ trusty/openstack-dashboard
2015-02-03 17:50:46,785 [DEBUG] deployer.charm: Branching charm lp:~openstack-charmers/charms/trusty/cinder/next @ trusty/cinder
2015-02-03 17:50:49,240 [DEBUG] deployer.charm: Branching charm lp:charms/quantum-gateway @ trusty/neutron-gateway
2015-02-03 17:50:51,875 [DEBUG] deployer.deploy: Resolving configuration
2015-02-03 17:50:51,984 [DEBUG] deployer.env: Connecting to environment...
2015-02-03 17:50:57,295 [ERROR] oil_ci.deploy.oil_deployer: Deployment failed:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/oil_ci/deploy/oil_deployer.py", line 140, in deploy
    debug=True, verbose=True)
  File "/usr/lib/python2.7/dist-packages/oil_ci/juju/juju_deployer.py", line 85, in run_deployer
    importer.Importer(env, deploy, options=opts).run()
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 194, in run
    self.env.connect()
  File "/usr/lib/python2.7/dist-packages/deployer/env/go.py", line 48, in connect
    self.client = EnvironmentClient.connect(self.name)
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 411, in connect
    return Connector().run(cls, env_name)
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 308, in run
    cls, address, data['user'], data['password'], cert_path)
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 313, in connect_env
    env.login(user="user-%s" % user, password=password)
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 159, in login
    "Params": {"AuthTag": user, "Password": password}})
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 140, in _rpc
    raw = self.conn.recv()
  File "/usr/lib/python2.7/dist-packages/websocket/_core.py", line 671, in recv
    opcode, data = self.recv_data()
  File "/usr/lib/python2.7/dist-packages/websocket/_core.py", line 686, in recv_data
    frame = self.recv_frame()
  File "/usr/lib/python2.7/dist-packages/websocket/_core.py", line 761, in recv_frame
    frame_buffer.recv_header(self._recv_strict)
  File "/usr/lib/python2.7/dist-packages/websocket/_core.py", line 265, in recv_header
    header = recv_fn(2)
  File "/usr/lib/python2.7/dist-packages/websocket/_core.py", line 868, in _recv_strict
    bytes = self._recv(shortage)
  File "/usr/lib/python2.7/dist-packages/websocket/_core.py", line 862, in _recv
    raise WebSocketConnectionClosedException()
WebSocketConnectionClosedException
2015-02-03 17:50:57,363 [INFO] oil_ci.deploy.oil_deployer: Saving deployment artifacts to: ./artifacts
2015-02-03 17:51:02,369 [ERROR] oil_ci.deploy.oil_deployer: Could not query juju status. Did environment successfully bootstrap? (Command '['timeout', '5', 'juju', 'status', '--format=json', '-e', u'ci-oil-slave10']' returned non-zero exit status 124)
+ rc=1
+ echo 'Deployment returned: 1'
Deployment returned: 1
+ [[ 1 == 0 ]]
+ test-catalog -p 03494fef-97ee-458b-83a4-bad35e3e3844 -t /etc/oil-ci/test-catalog-auth.json
2015-02-03 17:51:02 [INFO] test-catalog.: Loading system wide config: /etc/test-catalog/test-catalog.conf
+ oil-ci generate pipeline_tags --nodes=artifacts/oil_nodes --juju=artifacts/juju_status.yaml --out=artifacts/mapping.json
2015-02-03 17:51:03,677 [DEBUG] oil_ci: Loaded config from: /etc/oil-ci/oil-ci.conf
2015-02-03 17:51:03,927 [DEBUG] oil_ci.deploy.deployment: Creating loggers for deployer: ['', 'deploy.cli', 'deploy.env', 'deployer', 'deploy.deploy', 'deploy.charm', 'deploy.importer']
Command failed, please check log for more info
Traceback (most recent call last):
  File "/usr/bin/oil-ci", line 9, in <module>
    load_entry_point('canonical-oil==0.0.10', 'console_scripts', 'oil-ci')()
  File "/usr/lib/python2.7/dist-packages/oil_ci/cli/ci/main.py", line 196, in main
    return utils.run(sys.argv, categories)
  File "/usr/lib/python2.7/dist-packages/oil_ci/common/cliutils.py", line 118, in run
    ret = fn(*fn_args, **fn_kwargs)
  File "/usr/lib/python2.7/dist-packages/oil_ci/cli/ci/main.py", line 93, in pipeline_tags
    api.generate_pipeline_tags(nodes=nodes, juju=juju, out=out)
  File "/usr/lib/python2.7/dist-packages/oil_ci/cli/api.py", line 219, in generate_pipeline_tags
    mapping = HardwareMapper(nodes_f=nodes, juju_f=juju).get_service_map()
  File "/usr/lib/python2.7/dist-packages/oil_ci/cli/hardware_map.py", line 29, in __init__
    self.juju = load_file(juju_f)
  File "/usr/lib/python2.7/dist-packages/oil_ci/cli/hardware_map.py", line 20, in load_file
    with open(filename) as file_:
IOError: [Errno 2] No such file or directory: u'artifacts/juju_status.yaml'

[Kapil Thangavelu (hazmat)]

This has nothing to do with juju-deployer per se, even the cli isn't working, its clear the juju-core api server is not responding.

[Anastasia (anastasia-macmood)]

@Larry

Could you please provide machine and controller/state server logs?

[Anastasia (anastasia-macmood)]

Larry can no longer recreate.