Setup of MAAS Server using pod-cloud-installer failing

[Mark Wilkinson]

While installing a cloud using the pod-cloud installer version v0.11.1, seeing that it successfully installs the MAAS server, but continually fails trying to setup the lDS.

Here is the scenario:

1. Cabled 4 UCS-C servers properly for the networks as described in the pod installer PDF.
2. Setup deploy.cfg file.
3. Ran the pod-cloud-installer -c ~/deploy.cfg --install-maas successfully.
4. When running the pod-cloud-installer -c ~/deploy.cfg --install-lds, seeing consistent failures. Have tried working with others who have successfully set this up, and still cannot make this work.

Have tried running the following before running the --install-lDS, and it still doesn't work...

  sudo rm -f /etc/apt/sources.list.d/maas* /etc/apt/sources.list.d/juju*
  sudo apt-get update

It is very important that we get this setup to perform dev-test on components being delivered to the customer. Please reply ASAP with a solution.

Here is the error text from the install-lds portion of the log file:

2015-10-28 17:49:16,321 DEBUG Bootstrapping juju to vmLDS
2015-10-28 17:49:16,321 DEBUG CMD: ['juju', 'bootstrap', '-e', 'datacenter-maas', '--to', 'vmLDS.cloudvpn.com']
2015-10-28 18:03:02,602 DEBUG STDERR:
Bootstrapping environment "datacenter-maas"
Starting new instance for initial state server
Launching instance
 - /MAAS/api/1.0/nodes/node-90d75c48-7dbb-11e5-8bae-a0369f7f5c00/
Installing Juju agent on bootstrap instance
Waiting for address
Attempting to connect to vmLDS.cloudvpn.com:22
Attempting to connect to vmLDS.cloudvpn.com:22
Attempting to connect to 10.20.0.5:22
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@^M
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @^M
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@^M
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!^M
Someone could be eavesdropping on you right now (man-in-the-middle attack)!^M
It is also possible that a host key has just been changed.^M
The fingerprint for the ECDSA key sent by the remote host is
1e:fc:e8:cf:3a:28:7d:94:35:ea:ef:f9:36:f5:dd:a8.^M
Please contact your system administrator.^M
Add correct host key in /home/joswilki/.ssh/known_hosts to get rid of this message.^M
Offending ECDSA key in /home/joswilki/.ssh/known_hosts:1^M
  remove with: ssh-keygen -f "/home/joswilki/.ssh/known_hosts" -R 10.20.0.5^M
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.^M
Logging to /var/log/cloud-init-output.log on remote host
Running apt-get update
Running apt-get upgrade
Installing package: curl
Installing package: cpu-checker
Installing package: bridge-utils
Installing package: rsyslog-gnutls
Installing package: cloud-utils
Installing package: cloud-image-utils
Installing package: tmux
Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz <[https://streams.canonical.com/juju/tools/releases/juju-1.24.6-trusty-amd64.tgz]>
Bootstrapping Juju machine agent
Starting Juju machine agent (jujud-machine-0)
Bootstrap agent installed
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
ERROR upgrade in progress - Juju functionality is limited

2015-10-28 18:03:02,602 WARNING Hit error with juju bootstrap: Command '['juju', 'bootstrap', '-e', 'datacenter-maas', '--to', 'vmLDS.cloudvpn.com']' returned non-zero exit status 1
2015-10-28 18:03:02,602 DEBUG CMD: ['juju', 'status', '-e', 'datacenter-maas']
2015-10-28 18:03:12,796 DEBUG STDOUT:
environment: datacenter-maas
machines:
  "0":
    agent-state: pending
    agent-version: 1.24.6
    dns-name: vmLDS.cloudvpn.com
    instance-id: /MAAS/api/1.0/nodes/node-90d75c48-7dbb-11e5-8bae-a0369f7f5c00/
    series: trusty
    hardware: arch=amd64 cpu-cores=4 mem=8192M tags=virtual
    state-server-member-status: has-vote
services: {}

2015-10-28 18:03:12,798 DEBUG Checking juju status: [PASS]
2015-10-28 18:03:12,798 DEBUG Deploying Landscape to vmLDS
2015-10-28 18:03:12,806 DEBUG (cwd=/home/joswilki/.config/falkor/juju-home) RUN: ['juju-deployer', '-r', '180', '-vc', '/tmp/tmp0ohDel', '--timeout', '6000', '-e', 'datacenter-maas']
2015-10-28 18:03:12,806 DEBUG CMD: ['juju-deployer', '-r', '180', '-vc', '/tmp/tmp0ohDel', '--timeout', '6000', '-e', 'datacenter-maas']
2015-10-28 18:03:13,309 DEBUG STDERR:
2015-10-28 18:03:13 Using runtime GoEnvironment on datacenter-maas
2015-10-28 18:03:13 Retrying hooks for error resolution
2015-10-28 18:03:13 Connecting to environment...
2015-10-28 18:03:13 Connected to environment
Traceback (most recent call last):
  File "/usr/bin/juju-deployer", line 9, in <module>
    load_entry_point('juju-deployer==0.6.0', 'console_scripts', 'juju-deployer')()
  File "/usr/lib/python2.7/dist-packages/deployer/cli.py", line 135, in main
    run()
  File "/usr/lib/python2.7/dist-packages/deployer/cli.py", line 200, in run
    options.retry_count, watch=options.watch, timeout=options.timeout)
  File "/usr/lib/python2.7/dist-packages/deployer/env/go.py", line 188, in resolve_errors
    error_units = self._get_units_in_error()
  File "/usr/lib/python2.7/dist-packages/deployer/env/base.py", line 57, in _get_units_in_error
    status = self.status()
  File "/usr/lib/python2.7/dist-packages/deployer/env/go.py", line 233, in status
    return self.client.get_stat()
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 947, in get_stat
    watch = self.get_watch()
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 994, in get_watch
    watcher.start()
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 332, in start
    'Params': {}})
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 281, in _rpc
    raise EnvError(result)
jujuclient.EnvError: <Env Error - Details:
 { u'Error': u'upgrade in progress - Juju functionality is limited',
    u'RequestId': 0,
    u'Response': { }}
 >

[Mark Wilkinson (joswilki)]

Failure log added.

[Mark Wilkinson (joswilki)]

From looking at the comments for 1509097, I don't see a clear resolution path. My bug is also not an upgrade, but a fresh install.

 Should we be running our fresh install with the --no-auto-upgrade option given to the pod-cloud-installer command?

[David Britton (dpb)]

Hi Mark -- Please:

sudo apt-get update
sudo apt-get install juju-core

to make sure you are running 1.24.7 locally, this will workaround the issue.

I'll also make sure that the 0.11 series contains the fix so that no one else hits this.

[Mark Wilkinson (joswilki)]

Additionally, there is no explanation of what denotes a need for proxy in the instructions:

# Make sure this line is there if you need proxy support
Defaults env_keep="http_proxy https_proxy no_proxy"

How do we know if we need this? If we don't need it, does it mean we also should remove the proxy info from the deploy.cfg file?

    # The following optional proxy variables can be set if the machines to not
    # have direct internet access
    # Proxy url used for https traffic
    https-proxy: "http://proxy.esl.cisco.com:80"

    # Proxy url used for http traffic
    http-proxy: "http://proxy.esl.cisco.com:80"

Any clarification is appreciated,

Mark Wilkinson