socket.error: [Errno 110] Connection timed out while calling status()

[Larry Michel]

We've been seeing this periodically while deploying and sometimes more often:

2016-03-02 07:12:16 Service 'glance' does not need any more units added.
2016-03-02 07:12:16 Service 'swift-storage' does not need any more units added.
2016-03-02 07:12:16 Service 'rabbitmq-server' does not need any more units added.
2016-03-02 07:13:23 Config specifies num units for subordinate: ceilometer-agent
Traceback (most recent call last):
  File "/usr/bin/juju-deployer", line 9, in <module>
    load_entry_point('juju-deployer==0.6.2', 'console_scripts', 'juju-deployer')()
  File "/usr/lib/python2.7/dist-packages/deployer/cli.py", line 142, in main
    run()
  File "/usr/lib/python2.7/dist-packages/deployer/cli.py", line 241, in run
    importer.Importer(env, deployment, options).run()
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 431, in run
    self.add_units()
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 38, in add_units
    env_status = self.env.status()
  File "/usr/lib/python2.7/dist-packages/deployer/env/go.py", line 260, in status
    return self.client.get_stat()
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 1049, in get_stat
    watch = self.get_watch()
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 1075, in get_watch
    watch_env = Environment(self.endpoint)
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 522, in __init__
    self.conn = Connector.connect_socket(endpoint, self._ca_cert)
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 162, in connect_socket
    endpoint, origin=endpoint, sslopt=sslopt)
  File "/usr/lib/python2.7/dist-packages/websocket/_core.py", line 219, in create_connection
    websock.connect(url, **options)
  File "/usr/lib/python2.7/dist-packages/websocket/_core.py", line 438, in connect
    self.sock.connect(address)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 110] Connection timed out

These are current versions:

ii juju 1.25.3-0ubuntu1~14.04.1~juju1 all next generation service orchestration system
ii juju-core 1.25.3-0ubuntu1~14.04.1~juju1 amd64 Juju is devops distilled - client
ii juju-deployer 0.6.2.1-0ubuntu1~ppa1 amd64 A tool to deploy complex stacks of services using juju.
ii python-jujuclient 0.50.3+bzr69+30~ubuntu14.04.1 all Python API client for juju-core

[Cheryl Jennings (cherylj)]

Larry, can you attach machine-0.log?

[Larry Michel (lmic)]

Cheryl, the bootstrap node is no longer up, so I will have to wait for recreate.

[Larry Michel (lmic)]

Cheryl,
We're seeing a bunch of recreates but it's not possible to get the machine log. We are attempting to retrieve it but that's failing and the file is empty. It looks like the bootstrap node is unresponsive:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/oil_ci/deploy/oil_deployer.py", line 220, in deploy
    debug=True, verbose=True)
  File "/usr/lib/python2.7/dist-packages/oil_ci/juju/juju_deployer.py", line 89, in run_deployer
    importer.Importer(env, deploy, options=opts).run()
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 435, in run
    self.wait_for_units(ignore_errors=ignore_errors)
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 332, in wait_for_units
    services=self.deployment.get_service_names(), on_errors=on_errors)
  File "/usr/lib/python2.7/dist-packages/deployer/env/go.py", line 284, in wait_for_units
    services=services, on_errors=on_errors).run(callback)
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 2030, in run
    seen_initial = True
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 452, in __exit__
    self.stop()
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 432, in stop
    'Id': self.watcher_id})
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 313, in _rpc
    result = self._rpc_retry_if_upgrading(op)
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 326, in _rpc_retry_if_upgrading
    result = self._send_request(op)
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 339, in _send_request
    raw = self.conn.recv()
  File "/usr/lib/python2.7/dist-packages/websocket/_core.py", line 671, in recv
    opcode, data = self.recv_data()
  File "/usr/lib/python2.7/dist-packages/websocket/_core.py", line 686, in recv_data
    frame = self.recv_frame()
  File "/usr/lib/python2.7/dist-packages/websocket/_core.py", line 761, in recv_frame
    frame_buffer.recv_header(self._recv_strict)
  File "/usr/lib/python2.7/dist-packages/websocket/_core.py", line 265, in recv_header
    header = recv_fn(2)
  File "/usr/lib/python2.7/dist-packages/websocket/_core.py", line 868, in _recv_strict
    bytes = self._recv(shortage)
  File "/usr/lib/python2.7/dist-packages/websocket/_core.py", line 862, in _recv
    raise WebSocketConnectionClosedException()
WebSocketConnectionClosedException
2016-03-15 20:52:22,473 [INFO] oil_ci.deploy.oil_deployer: Saving deployment artifacts to: ./artifacts
2016-03-15 20:53:06,912 [ERROR] oil_ci.deploy.oil_deployer: Could not query juju status. Did environment successfully bootstrap?
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/oil_ci/deploy/oil_deployer.py", line 125, in save_juju_status
    status = client.status(environment=self.env_name, timeout=5)
  File "/usr/lib/python2.7/dist-packages/oil_ci/juju/client.py", line 88, in status
    output = check_output(args, stderr=open("/dev/null"))
  File "/usr/lib/python2.7/subprocess.py", line 544, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['timeout', '5', 'juju', 'status', '--format=json', '-e', u'ci-o...

[Cheryl Jennings (cherylj)]

Larry, lacking machine-0.log, can you give a recreate scenario?

Juju / MAAS levels? Bundle being deployed? Anything else special about the environment?

[berkninan (berkninan)]

If you have chosen a random, but valid, IP address and port, socket.connect() will attempt to make a connection to that endpoint. By default, if no explicit timeout is set for the socket, it will block while doing so and eventually timeout, raising exception socket.error: [Errno 110] Connection timed out. The default timeout on my machine is 120 seconds.

http://net-informations.com/python/net/socket.htm

[Tom Haddon (mthaddon)]

Marking as won't fix given the age of the bug and that this refers to Juju 1.x. Also, presumably we'd use native juju bundles for this now. Please let us know if that's incorrect.