debug-log does not work with local provider

[Stuart Bishop]

This is a regression of Bug #1202682

juju debug-log does not work with the local provider with juju 1.24 beta 6

stub@aargh:~$ juju debug-log
ERROR cannot open log file: open /var/log/juju-stub-local/all-machines.log: no such file or directory
stub@aargh:~$ juju --version
1.24-beta6-vivid-amd64

[Curtis Hovey (sinzui)]

Note that bug 1396159 reports only machine-0 is in the log. This appears to be an moved log or permission change.

[Menno Finlay-Smits (menno.smits)]

The problem is that the location of rsyslog's certificate files has changed recently. They used to be in the log directory but have now been moved to the data directory. For the local provider this is will probably be ~/.juju/local. If this directory, or ~/.juju or ~ is not world accessible (.juju was 0700 on my machine) then rsyslogd can't get to the cert files and fails.

You can see the errors from rsyslogd in /var/log/syslog.

The best fix is probably to move these files to world-readable Juju directory. /var/lib/juju/local/<namespace> has been suggested.

[Eric Snow (ericsnowcurrently)]

related: lp:1370896

[Eric Snow (ericsnowcurrently)]

I've duplicated the issue using the local provider and verified that debug-log works correctly using the "ec2" provider.

[Eric Snow (ericsnowcurrently)]

I'm not sure that the location of the config/cert files is a factor here. The all-machines.log file really *does not* exist. In contrast, the machine-0.log file *does* exist. I'll look into why local provider is different.

[Eric Snow (ericsnowcurrently)]

Ah. From /var/log/syslog:

  Jun 17 13:08:21 ... rsyslogd-2040: can not read file '/home/esnow/.juju/local/rsyslog-cert.pem' [try http://www.rsyslog.com/e/2040 ]

So the new location *is* causing a problem on local provider. The rsyslog user cannot reach the file due to the permissions on ~/.juju.

Now back to how to get it to work...

[Eric Snow (ericsnowcurrently)]

Also, if you set the root-dir option for local provider in environments.yaml to something syslog can read then debug-log doesn't have any trouble. I expect the same thing is true if you set the JUJU_HOME env var to an existing directory.

[Eric Snow (ericsnowcurrently)]

Possible solutions:

* set permissions for ~/.juju to 0777 (WriteEnvirons in environs/config.go)
* set permissions for ~/.juju to 0770 and set the group to "syslog"
* change the default root dir for local provider to /var/lib/juju-<env name>
* add symlinks for the files to a place that the syslog user can read them
* keep the rsyslog files in /var/lib/juju no matter what

I'm favoring that last one.

[Eric Snow (ericsnowcurrently)]

Those group permissions should be 5 rather than 7.

[Eric Snow (ericsnowcurrently)]

http://reviews.vapour.ws/r/1963/

[Eric Snow (ericsnowcurrently)]

The 1.24 needs to be merged into master.

[lithium (rudicba)]

This is still failing in juju 1.24.2

How to reproduce:

### Bootstrap and test

$ juju switch local
$ juju bootstrap
$ juju debug-log (works fine)

### Destroy environment and create again

$ juju destroy-environment local
$ juju bootstrap
$ juju debug-log

ERROR cannot open log file: open /var/log/juju-name-local/all-machines.log: no such file or directory

[lithium (rudicba)]

Need to change local environment name every time is destroyed for use debug-log

[Andrew Wilkins (axwalk)]

I'm looking into this latest issue with debug-log in the local provider. I have an idea about what's going on; I think the worker is not publishing the CA cert to state, because it's already on disk when the agent starts up.

[Andrew Wilkins (axwalk)]

https://github.com/juju/juju/pull/2804 fixes the issue reported by lithium

[Ryan Harper (raharper)]

Broken for me on Xenial amd64 with 1.25.0

I'm in a Xenial lxd container:

apt-get install juju-local
su - ubuntu
juju init
juju switch local
juju bootstrap
juju debug-log

ERROR

ubuntu@nestc1:~$ juju init
A boilerplate environment configuration file has been written to /home/ubuntu/.juju/environments.yaml.
Edit the file to configure your juju environment and run bootstrap.
ubuntu@nestc1:~$ juju switch local
-> local
ubuntu@nestc1:~$ juju bootstrap
Bootstrapping environment "local"
Starting new instance for initial state server
Building tools to upload (1.25.0.1-xenial-amd64)
Installing Juju agent on bootstrap instance
sudo: unable to resolve host nestc1
Logging to /home/ubuntu/.juju/local/cloud-init-output.log on remote host
Installing package: curl
Installing package: cpu-checker
Installing package: bridge-utils
Installing package: rsyslog-gnutls
Installing package: cloud-utils
Installing package: cloud-image-utils
Installing package: tmux
Bootstrapping Juju machine agent
Starting Juju machine agent (juju-agent-ubuntu-local)
Bootstrap agent installed
Waiting for API to become available
Waiting for API to become available
Bootstrap complete
ubuntu@nestc1:~$ juju status
environment: local
machines:
  "0":
    agent-state: started
    agent-version: 1.25.0.1
    dns-name: localhost
    instance-id: localhost
    series: xenial
    state-server-member-status: has-vote
services: {}
ubuntu@nestc1:~$ juju deploy ubuntu
Added charm "cs:trusty/ubuntu-5" to the environment.
ubuntu@nestc1:~$ juju debug-log
ERROR cannot open log file: open /var/log/juju-ubuntu-local/all-machines.log: no such file or directory

[Cheryl Jennings (cherylj)]

Well that's unfortunate.

Ryan, can you open a new bug and attach /var/log/syslog?

[Ryan Harper (raharper)]

Opened https://bugs.launchpad.net/juju-core/+bug/1531589 and attached
/var/log and /var/lib/juju

On Tue, Jan 5, 2016 at 11:50 AM, Cheryl Jennings <
<email address hidden>> wrote:

> Well that's unfortunate.
>
> Ryan, can you open a new bug and attach /var/log/syslog?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1464335
>
> Title:
> debug-log does not work with local provider
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1464335/+subscriptions
>