Juju2 with MAAS2, log shows errors after having created a controller

Bug #1592179 reported by Peter Matulis
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
juju-core
Expired
Undecided
Unassigned

Bug Description

On a fresh Xenial system I am using Juju 2.0 and MAAS 2.0. After creating a controller, which seemed to go fine, I am getting errors in the logs. Both Juju and MAAS are on the same host. The MAAS nodes (the one used for the controller) is a KVM guest on another system.

Juju client: 192.168.1.5
MAAS server: 192.168.1.5
Juju controller: 192.168.1.64

$ export JUJU_DEV_FEATURE_FLAGS=maas2
$ juju bootstrap --bootstrap-constraints arch=amd64 maas-controller golem-maas

$ juju list-controllers

CONTROLLER MODEL USER CLOUD/REGION
local.maas-controller* controller admin@local golem-maas

$ juju show-controller maas-controller
====================================
maas-controller:
  details:
    uuid: b3073ded-8221-4d42-852f-50c69234dcf8
    api-endpoints: ['192.168.1.64:17070']
    ca-cert: |
      -----BEGIN CERTIFICATE-----
      MIID0zCCArugAwIBAgIUCCa6FtYBzQrgSLIkbUYqh1pUXTIwDQYJKoZIhvcNAQEL
      BQAwcTENMAsGA1UEChMEanVqdTExMC8GA1UEAwwoanVqdS1nZW5lcmF0ZWQgQ0Eg
      Zm9yIG1vZGVsICJjb250cm9sbGVyIjEtMCsGA1UEBRMkYjMwNzNkZWQtODIyMS00
      ZDQyLTg1MmYtNTBjNjkyMzRkY2Y4MB4XDTE2MDYwNjIwMzIxMFoXDTI2MDYxMzIw
      MzIwOVowcTENMAsGA1UEChMEanVqdTExMC8GA1UEAwwoanVqdS1nZW5lcmF0ZWQg
      Q0EgZm9yIG1vZGVsICJjb250cm9sbGVyIjEtMCsGA1UEBRMkYjMwNzNkZWQtODIy
      MS00ZDQyLTg1MmYtNTBjNjkyMzRkY2Y4MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
      MIIBCgKCAQEAqIIcvIKGeai3Pq9Wy31zbrVGKae+tZTYw6CQk8RIyVdNnxewodjc
      fg+fiZaZEWknqJR26bQLsKccjVVpyJzG/HyRAK4N6m6JfSnwRvhS2EYrgFIsaIvx
      fuA8oU+57F5QgjO19NfGs7Rhn5PikacByaZXHbkTi6LfVtKTmpEkvnA/zgUl+CIQ
      I9MoKXAVf6LGt17kOFbU71dRJG5zQ0AR8bP+UAkBeTF0RQjRlPnQV3A/GvI6GTyi
      rZn/XxYMVKFYHlrMTAAeXPTq+rK5UHYNzaQlGvsGyjbA3iqEKwSeNyrL0MajkKSr
      snoPOOF/2OcHS/g7DaiyiY099AcNMQ9/mwIDAQABo2MwYTAOBgNVHQ8BAf8EBAMC
      AqQwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUy0vGEvcUwVMD5dHRPaxYVhIW
      i+AwHwYDVR0jBBgwFoAUy0vGEvcUwVMD5dHRPaxYVhIWi+AwDQYJKoZIhvcNAQEL
      BQADggEBAHr+Uy+UACWYzs/PsgPSsHWnSd2pcPXyffqC2huIsHHQxSV+k0OajQnh
      zK3Ec1U7SljP81aG0ebWl9A473VrIh5mwk28p6t4nysqgLYeOoWPtQLnn24wfZxJ
      PQMH8ThVxVtvVznODITSgSFFSMD6FPJEksEFpGg841a/oHug4o54yMGc1aArWdye
      KIbnuOpq/MdOBMdbYnqpojqDzzdMXAOURbQ0/0FQdi8RLRDKiGZwyL1oVskEtXmM
      NrZju2r6i5u/m7HhvOL4+glMZzbyBQgyIk5rEF41/LhR/NjrJYoGJvfnNMNsY8KJ
      3mOwVK55Fap6Qa550bS6Wv9ZjR5puic=
      -----END CERTIFICATE-----
    cloud: golem-maas
  accounts:
    admin@local:
      user: admin@local
      models:
        controller:
          uuid: b3073ded-8221-4d42-852f-50c69234dcf8
        default:
          uuid: ff5a5833-6979-4129-849b-c0b9d6b03892
      current-model: controller
  current-account: admin@local
  bootstrap-config:
    cloud: golem-maas
    cloud-type: maas
    endpoint: http://192.168.1.5/MAAS
    credential: golem
====================================

$ juju switch

local.maas-controller:controller

$ juju debug-log --replay -T -l TRACE | grep ERROR

machine-0: 2016-06-13 20:42:14 ERROR juju.worker.dependency engine.go:526 "api-caller" manifold worker returned unexpected error: cannot open api: unable to connect to API: websocket.Dial wss://localhost:17070/mo
del/b3073ded-8221-4d42-852f-50c69234dcf8/api: dial tcp 127.0.0.1:17070: getsockopt: connection refused machine-0: 2016-06-13 20:42:18 ERROR juju.api.watcher watcher.go:86 error trying to stop watcher: connection is shut down
machine-0: 2016-06-13 20:42:18 ERROR juju.api.watcher watcher.go:86 error trying to stop watcher: connection is shut down
machine-0: 2016-06-13 20:42:18 ERROR juju.api.watcher watcher.go:86 error trying to stop watcher: connection is shut down
machine-0: 2016-06-13 20:42:18 ERROR juju.api.watcher watcher.go:86 error trying to stop watcher: connection is shut down
machine-0: 2016-06-13 20:42:18 ERROR juju.api.watcher watcher.go:86 error trying to stop watcher: connection is shut down
machine-0: 2016-06-13 20:42:18 ERROR juju.api.watcher watcher.go:86 error trying to stop watcher: connection is shut down
machine-0: 2016-06-13 20:42:18 ERROR juju.worker.logger logger.go:47 connection is shut down
machine-0: 2016-06-13 20:42:18 ERROR juju.rpc server.go:577 error writing response: write tcp 127.0.0.1:17070->127.0.0.1:45216: write: connection reset by peer
machine-0: 2016-06-13 20:42:18 ERROR juju.rpc server.go:577 error writing response: write tcp 127.0.0.1:17070->127.0.0.1:45216: write: connection reset by peer
machine-0: 2016-06-13 20:42:18 ERROR juju.rpc server.go:577 error writing response: write tcp 127.0.0.1:17070->127.0.0.1:45216: write: connection reset by peer
machine-0: 2016-06-13 20:42:18 ERROR juju.rpc server.go:577 error writing response: write tcp 127.0.0.1:17070->127.0.0.1:45216: write: connection reset by peer
machine-0: 2016-06-13 20:42:18 ERROR juju.rpc server.go:577 error writing response: write tcp 127.0.0.1:17070->127.0.0.1:45216: write: connection reset by peer
machine-0: 2016-06-13 20:42:18 ERROR juju.rpc server.go:577 error writing response: write tcp 127.0.0.1:17070->127.0.0.1:45216: write: connection reset by peer
machine-0: 2016-06-13 20:42:18 ERROR juju.rpc server.go:577 error writing response: write tcp 127.0.0.1:17070->127.0.0.1:45216: write: connection reset by peer
machine-0: 2016-06-13 20:42:18 ERROR juju.rpc server.go:577 error writing response: write tcp 127.0.0.1:17070->127.0.0.1:45216: write: connection reset by peer
machine-0: 2016-06-13 20:42:18 ERROR juju.rpc server.go:577 error writing response: write tcp 127.0.0.1:17070->127.0.0.1:45216: write: connection reset by peer
machine-0: 2016-06-13 20:42:18 ERROR juju.rpc server.go:577 error writing response: write tcp 127.0.0.1:17070->127.0.0.1:45216: write: connection reset by peer
machine-0: 2016-06-13 20:42:18 ERROR juju.rpc server.go:577 error writing response: write tcp 127.0.0.1:17070->127.0.0.1:45216: write: connection reset by peer
machine-0: 2016-06-13 20:42:18 ERROR juju.rpc server.go:577 error writing response: write tcp 127.0.0.1:17070->127.0.0.1:45216: write: connection reset by peer
machine-0: 2016-06-13 20:42:18 ERROR juju.rpc server.go:577 error writing response: write tcp 127.0.0.1:17070->127.0.0.1:45216: write: connection reset by peer
machine-0: 2016-06-13 20:42:18 ERROR juju.rpc server.go:577 error writing response: write tcp 127.0.0.1:17070->127.0.0.1:45216: write: connection reset by peer
machine-0: 2016-06-13 20:42:18 ERROR juju.rpc server.go:577 error writing response: write tcp 127.0.0.1:17070->127.0.0.1:45216: write: connection reset by peer
machine-0: 2016-06-13 20:42:18 ERROR juju.rpc server.go:577 error writing response: write tcp 127.0.0.1:17070->127.0.0.1:45216: write: connection reset by peer
machine-0: 2016-06-13 20:42:21 ERROR juju.rpc server.go:577 error writing response: write tcp 127.0.0.1:17070->127.0.0.1:45216: write: connection reset by peer
machine-0: 2016-06-13 20:54:12 ERROR juju.apiserver.presence.machine-0 pinger.go:207 pinger failed: E11000 duplicate key error collection: presence.presence.pings index: _id_ dup key: { : "b3073ded-8221-4d42-852f
-50c69234dcf8:1465851240" }

description: updated
Revision history for this message
Peter Matulis (petermatulis) wrote :

Here is /var/log/syslog from the controller:

http://paste.ubuntu.com/17311172/

Revision history for this message
Cheryl Jennings (cherylj) wrote :

There wasn't anything that stood out in the syslog. Do you have the complete machine-0.log from the controller?

Changed in juju-core:
status: New → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for juju-core because there has been no activity for 60 days.]

Changed in juju-core:
status: Incomplete → Expired
Revision history for this message
Asad Saeeduddin (masaeedu) wrote :
Download full text (94.3 KiB)

Still seeing this with:
 - juju 2.0.0-xenial-amd64
 - lxd 2.0.8
 - Xenial 16.04.1 LTS

On the controller, /var/log/juju/machine-0.log appears to have the same contents as the juju debug-log output described in the issue, but I'm reproducing the contents here nevertheless:

ubuntu@juju-8c5551-0:~$ sudo cat /var/log/juju/machine-0.log
2017-02-01 01:58:15 INFO juju.cmd supercommand.go:63 running jujud [2.0.0 gc go1.6.2]
2017-02-01 01:58:15 DEBUG juju.cmd supercommand.go:64 args: []string{"/var/lib/juju/tools/machine-0/jujud", "machine", "--data-dir", "/var/lib/juju", "--machine-id", "0", "--debug"}
2017-02-01 01:58:15 DEBUG juju.agent agent.go:509 read agent config, format "2.0"
2017-02-01 01:58:15 INFO juju.cmd.jujud machine.go:404 machine agent machine-0 start (2.0.0 [gc])
2017-02-01 01:58:16 DEBUG juju.wrench wrench.go:112 couldn't read wrench directory: stat /var/lib/juju/wrench: no such file or directory
2017-02-01 01:58:16 INFO juju.worker.upgradesteps worker.go:72 no upgrade steps required or upgrade steps for 2.0.0 have already been run.
2017-02-01 01:58:16 INFO juju.worker runner.go:256 start "engine"
2017-02-01 01:58:16 DEBUG juju.worker.dependency engine.go:492 "tools-version-checker" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2017-02-01 01:58:16 DEBUG juju.worker.dependency engine.go:492 "machine-action-runner" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2017-02-01 01:58:16 DEBUG juju.worker.dependency engine.go:478 "termination-signal-handler" manifold worker started
2017-02-01 01:58:16 DEBUG juju.worker.dependency engine.go:492 "state-config-watcher" manifold worker stopped: "agent" not running: dependency not available
2017-02-01 01:58:16 DEBUG juju.worker.dependency engine.go:478 "upgrade-check-gate" manifold worker started
2017-02-01 01:58:16 DEBUG juju.worker.dependency engine.go:492 "api-config-watcher" manifold worker stopped: "agent" not running: dependency not available
2017-02-01 01:58:16 DEBUG juju.worker.dependency engine.go:492 "proxy-config-updater" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2017-02-01 01:58:16 DEBUG juju.worker.dependency engine.go:492 "reboot-executor" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2017-02-01 01:58:16 DEBUG juju.worker.dependency engine.go:492 "disk-manager" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2017-02-01 01:58:16 DEBUG juju.worker.dependency engine.go:492 "log-forwarder" manifold worker stopped: "upgrade-check-flag" not running: dependency not available
2017-02-01 01:58:16 DEBUG juju.worker.dependency engine.go:492 "api-address-updater" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2017-02-01 01:58:16 DEBUG juju.worker.dependency engine.go:492 "machiner" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2017-02-01 01:58:16 DEBUG juju.worker.dependency engine.go:492 "ssh-identity-writer" manifold worker stopped: "migration-inactive-flag" not running: dependency not available...

Revision history for this message
Asad Saeeduddin (masaeedu) wrote :

Sorry, not sure if previous comment was snipped. Here it is again as an attachment.

Revision history for this message
Anastasia (anastasia-macmood) wrote :

Thank you for attaching the log \o/

Could you please describe what you are bootstrapping - command arguments, configuration options?
Could you please provide log with TRACE enabled? It might shed more light on what has happened.

Also it is not clear from given information where the environment recovered or whether it is now unusable for you.

Revision history for this message
Asad Saeeduddin (masaeedu) wrote :

I am following the instructions here: https://insights.ubuntu.com/2016/06/08/lxd-2-0-lxd-and-juju/ to get juju up and running with a local LXD cloud in a Ubuntu 16.04 desktop Hyper-V VM.

Here are the steps I followed on first boot:

 - Install lxd, zfs utils
 - Run `sudo lxd init` and accept defaults for all options **except** IPv6
 - Reboot to avoid any issues with groups
 - Run `sudo apt-get install juju -y`
 - Run `juju bootstrap`, accept the default "localhost" as the cloud and the default "localhost-localhost" as the name of the controller
 - As soon as container provisioning completes and the `juju bootstrap` command exits, run `juju switch controller`, followed by `juju debug-log --replay`

This produces log output containing the errors I posted.

I arrived at this issue while trying to diagnose why invoking `juju deploy <application>` results in containers stuck in pending state, and no output in the "default" model's `juju debug-log`. I spotted the port 17070 errors in the "controller" model's log and tried setting up from scratch, and found that I get errors in the log before ever trying to deploy any applications. I am not sure if this is the root cause of the issue I'm seeing, but it can't hurt to try and settle this bug one way or another.

I am not at my computer at the moment but will provide the extra information you requested some time tomorrow.

Revision history for this message
Asad Saeeduddin (masaeedu) wrote :

@anastasia I've been looking at the docs, but can't figure out how I would enable TRACE. Could you point me in the right direction?

Revision history for this message
Anastasia (anastasia-macmood) wrote :

Something along the lines of
`JUJU_LOGGING_CONFIG="<root>=TRACE" juju bootstrap ...`

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.