plumgrid director fails to come up

[Jason Hobbs]

This is related to bug 1515049, but this bug is about it not working, rather than the failure not being reported.

plumgrid-director is not coming up, resulting in neutron-server not being able to reach it:

jenkins@jason-slave:~$ neutron security-group-list
PLUMgrid Plugin Error: Error connection to HTTPS server (HTTP 500) (Request-ID: req-157d9df1-5bc5-4c7e-b35e-cdda00baca1c)

Here is the bundle:
http://paste.ubuntu.com/18658630/

Here is the juju status:
http://paste.ubuntu.com/18660611/

This is with juju 2.0 beta 11.

[Junaid Ali (junaidali)]

Jason, can you please attach neutron server logs as well?

[Jason Hobbs (jason-hobbs)]

Hi Junaid,

Attached.

On Thu, Jul 7, 2016 at 1:09 AM, Junaid Ali <email address hidden> wrote:

> Jason, can you please attach neutron server logs as well?
>
> ** Changed in: plumgrid-director (Juju Charms Collection)
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1599640
>
> Title:
> plumgrid director fails to come up
>
> Status in plumgrid-director package in Juju Charms Collection:
> Incomplete
>
> Bug description:
> This is related to bug 1515049, but this bug is about it not working,
> rather than the failure not being reported.
>
> plumgrid-director is not coming up, resulting in neutron-server not
> being able to reach it:
>
> jenkins@jason-slave:~$ neutron security-group-list
> PLUMgrid Plugin Error: Error connection to HTTPS server (HTTP 500)
> (Request-ID: req-157d9df1-5bc5-4c7e-b35e-cdda00baca1c)
>
> Here is the bundle:
> http://paste.ubuntu.com/18658630/
>
> Here is the juju status:
> http://paste.ubuntu.com/18660611/
>
> This is with juju 2.0 beta 11.
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/charms/+source/plumgrid-director/+bug/1599640/+subscriptions
>

[Ashley Lai (alai)]

Bilal and I spent a bit of time to debug this and found that the plumgrid services did not start and therefore the license could not get installed. After started the services and updated the license, we were able to create the network. Below is the debug log:

juju log: http://paste.ubuntu.com/18769850/

ubuntu@hayward-42:~$ ifconfig br-eth0:1
br-eth0:1 Link encap:Ethernet HWaddr 00:22:99:e0:03:47
         UP BROADCAST RUNNING MULTICAST MTU:1580 Metric:1

ubuntu@hayward-42:~$ cat /var/run/plumgrid/lxc/ifc_list_gateway
cat: /var/run/plumgrid/lxc/ifc_list_gateway: No such file or directory
ubuntu@hayward-42:~$ ls /var/run/plumgrid/lxc
ubuntu@hayward-42:~$ sudo su
root@hayward-42:/home/ubuntu# ls /var/run/plumgrid/lxc
root@hayward-42:/home/ubuntu# ps aux | grep launch
root 10149 0.0 0.0 10460 932 pts/7 S+ 07:46 0:00 grep --color=auto launch
root@hayward-42:/home/ubuntu#

# virsh -c lxc: list --all
Id Name State
----------------------------------------------------
1977 plumgrid running

root@hayward-42:/var/log/juju# uname -a
Linux hayward-42 3.13.0-91-generic #138-Ubuntu SMP Fri Jun 24 17:00:34 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
root@hayward-42:/var/log/juju# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty

root@hayward-42:/var/log/juju# dpkg -l | grep libvirt
ii libvirt-bin 1.2.2-0ubuntu13.1.17 amd64 programs for the libvirt library
ii libvirt0 1.2.2-0ubuntu13.1.17 amd64 library for interfacing with different virtualization systems
root@hayward-42:/var/log/juju#

$ neutron net-create ashley
500-{u'NeutronError': {u'message': u'PLUMgrid Plugin Error: PLUMgrid Platform REST Error = "Internal error (/connectivity/domain): Unknown license item name VND"', u'type': u'PLUMgridException', u'detail': u''}}

[Junaid Ali (junaidali)]

Ashley/Jason, please let us know if this issue hits again.

[Jason Hobbs (jason-hobbs)]

Junaid,

Was there a fix made for this in the plumgrid-director charm?

[Junaid Ali (junaidali)]

Jason,
I did a deployment with Juju 2.0 but couldn't reproduce this issue, so it seems that somehow plumgrid service didn't come up on your setup. Let me try out your bundle.

[Junaid Ali (junaidali)]

SOrry for the late response, I was not subscribed to "all notifications" for this bug.

[Junaid Ali (junaidali)]

Ashley, I have reproduced the issue with the bundle you shared. Looking into the issue

[Junaid Ali (junaidali)]

Director charm for custom-lxc package fixes this issue. This will be automatically fixed once docker director charm is released, as this is a packaging issue.