debug-log is unstable with LXD local provider

[Peter Matulis]

$ dpkg-query -W juju-local lxd
juju-local 1.26-alpha3-0ubuntu1~15.10.1~juju1
lxd 0.26-0ubuntu3~ubuntu15.10.1~ppa1

$ uname -a
Linux pmatulis-lxd1 4.2.0-23-generic #28-Ubuntu SMP Sun Dec 27 17:47:31 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -c
Codename: wily

After bootstrapping everything seems ok:

$ juju debug-log
machine-0: 2016-01-15 19:34:36 DEBUG juju.apiserver apiserver.go:275 <- [4] machine-0 {"RequestId":25,"Type":"Machiner","Request":"SetMachineAddresses","Params":"'params redacted'"}
machine-0: 2016-01-15 19:34:36 DEBUG juju.apiserver apiserver.go:275 <- [4] machine-0 {"RequestId":26,"Type":"StorageProvisioner","Version":1,"Request":"WatchBlockDevices","Params":"'params redacted'"}
machine-0: 2016-01-15 19:34:36 DEBUG juju.apiserver apiserver.go:275 <- [4] machine-0 {"RequestId":27,"Type":"NotifyWatcher","Id":"4","Request":"Next","Params":"'params redacted'"}
machine-0: 2016-01-15 19:34:36 DEBUG juju.apiserver apiserver.go:289 -> [4] machine-0 10.180489ms {"RequestId":16,"Response":"'body redacted'"} Logger[""].LoggingConfig
machine-0: 2016-01-15 19:34:36 DEBUG juju.worker.logger logger.go:45 reconfiguring logging from "<root>=DEBUG" to "<root>=WARNING;unit=DEBUG"
machine-0: 2016-01-15 19:34:36 WARNING juju.cmd.jujud machine.go:1015 determining kvm support: INFO: /dev/kvm does not exist
machine-0: HINT: sudo modprobe kvm_intel
machine-0: modprobe: ERROR: ../libkmod/libkmod.c:556 kmod_search_moddep() could not open moddep file '/lib/modules/4.2.0-23-generic/modules.dep.bin'
machine-0: : exit status 1
machine-0: no kvm containers possible

But seems to break down after having deployed some simple machines:

$ juju status
environment: lxd
machines:
  "0":
    agent-state: started
    agent-version: 1.26-alpha3.1
    dns-name: 10.0.3.121
    instance-id: juju-5ef5dcb0-776b-4119-8e6f-2f6f6dfc29d9-machine-0
    instance-state: Running
    series: trusty
    hardware: arch=amd64 cpu-cores=0 mem=0M
    state-server-member-status: has-vote
  "1":
    agent-state: started
    agent-version: 1.26-alpha3.1
    dns-name: 10.0.3.63
    instance-id: juju-5ef5dcb0-776b-4119-8e6f-2f6f6dfc29d9-machine-1
    instance-state: Running
    series: trusty
    hardware: arch=amd64 cpu-cores=0 mem=0M
  "2":
    agent-state: started
    agent-version: 1.26-alpha3.1
    dns-name: 10.0.3.208
    instance-id: juju-5ef5dcb0-776b-4119-8e6f-2f6f6dfc29d9-machine-2
    instance-state: Running
    series: trusty
    hardware: arch=amd64 cpu-cores=0 mem=0M
services:
  mysql:
    charm: cs:trusty/mysql-33
    exposed: false
    service-status:
      current: unknown
      since: 15 Jan 2016 19:44:45Z
    relations:
      cluster:
      - mysql
    units:
      mysql/0:
        workload-status:
          current: unknown
          since: 15 Jan 2016 19:44:45Z
        agent-status:
          current: idle
          since: 15 Jan 2016 19:44:46Z
          version: 1.26-alpha3.1
        agent-state: started
        agent-version: 1.26-alpha3.1
        machine: "2"
        public-address: 10.0.3.208
  ubuntu:
    charm: cs:trusty/ubuntu-5
    exposed: false
    service-status:
      current: unknown
      since: 15 Jan 2016 19:38:12Z
    units:
      ubuntu/0:
        workload-status:
          current: unknown
          since: 15 Jan 2016 19:38:12Z
        agent-status:
          current: idle
          since: 15 Jan 2016 19:42:14Z
          version: 1.26-alpha3.1
        agent-state: started
        agent-version: 1.26-alpha3.1
        machine: "1"
        public-address: 10.0.3.63

$ juju debug-log
machine-0: 2016-01-15 19:34:36 DEBUG juju.apiserver apiserver.go:275 <- [4] machine-0 {"RequestId":26,"Type":"StorageProvisioner","Version":1,"Request":"WatchBlockDevices","Params":"'params redacted'"}
machine-0: 2016-01-15 19:34:36 DEBUG juju.apiserver apiserver.go:275 <- [4] machine-0 {"RequestId":27,"Type":"NotifyWatcher","Id":"4","Request":"Next","Params":"'params redacted'"}
machine-0: 2016-01-15 19:34:36 DEBUG juju.apiserver apiserver.go:289 -> [4] machine-0 10.180489ms {"RequestId":16,"Response":"'body redacted'"} Logger[""].LoggingConfig
machine-0: 2016-01-15 19:34:36 DEBUG juju.worker.logger logger.go:45 reconfiguring logging from "<root>=DEBUG" to "<root>=WARNING;unit=DEBUG"
machine-0: 2016-01-15 19:34:36 WARNING juju.cmd.jujud machine.go:1015 determining kvm support: INFO: /dev/kvm does not exist
machine-0: HINT: sudo modprobe kvm_intel
machine-0: modprobe: ERROR: ../libkmod/libkmod.c:556 kmod_search_moddep() could not open moddep file '/lib/modules/4.2.0-23-generic/modules.dep.bin'
machine-0: : exit status 1
machine-0: no kvm containers possible

machine-0: 2016-01-15 19:35:46 ERROR juju.apiserver apiserver.go:443 error serving RPCs: error receiving message: read tcp 10.0.3.121:17070->10.0.3.1:38726: read: connection reset by peer

Ctrl-c

$ juju debug-log
ERROR unable to read initial response: EOF

Then it seems to work again:

$ juju debug-log
machine-0: github.com/juju/juju/apiserver.(*debugLogHandler).ServeHTTP(0xc820210a00, 0x7f200a805950, 0xc8214ce160, 0xc820a8f6c0)
machine-0: #011/build/juju-core-gln3LD/juju-core-1.26-alpha3/src/github.com/juju/juju/apiserver/debuglog.go:101 +0xc8
machine-0: github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP(0xc82056a048, 0x7f200a805950, 0xc8214ce160, 0xc820a8f6c0)
machine-0: #011/build/juju-core-gln3LD/juju-core-1.26-alpha3/src/github.com/bmizerany/pat/mux.go:109 +0x244
machine-0: net/http.serverHandler.ServeHTTP(0xc8204df800, 0x7f200a805950, 0xc8214ce160, 0xc820a8f6c0)
machine-0: #011/usr/lib/go/src/net/http/server.go:1862 +0x19e
machine-0: net/http.(*conn).serve(0xc8214c5c30)
machine-0: #011/usr/lib/go/src/net/http/server.go:1361 +0xbee
machine-0: created by net/http.(*Server).Serve
machine-0: #011/usr/lib/go/src/net/http/server.go:1910 +0x3f6

[Peter Matulis (petermatulis)]

Possibly related:

https://bugs.launchpad.net/juju-core/+bug/1531589
https://bugs.launchpad.net/juju-core/+bug/1202682
https://bugs.launchpad.net/juju-core/+bug/1390585
https://bugs.launchpad.net/juju-core/+bug/1491688

[Cheryl Jennings (cherylj)]

Peter - are you still running into this issue?

[Stuart Bishop (stub)]

I filed a dupe, and it seems fixed to me now.

[Peter Matulis (petermatulis)]

@Cheryl
I can't even get Juju to work with LXD at this time.

[Launchpad Janitor (janitor)]

[Expired for juju-core because there has been no activity for 60 days.]