neutron-server fails to start; python-neutron-vpnaas and python-neutron-lbaas packages are missing.

[Syed Armani]

When deploying OpenStack Kilo release neutron-server will fail to start. Initiallly the neuron-server logs will suggest
that db table 'neutron.ml2_gre_allocations' doesn't exist.

root@alice:/var/log/neutron# tail -n100 neutron-server.log
2015-06-25 21:28:33.570 9129 TRACE neutron.service OperationalError: (OperationalError) no such table: ml2_vlan_allocations u'SELECT ml2_vlan_allocations.physical_network AS ml2_vlan_allocations_physical_network, ml2_vlan_allocations.vlan_id AS ml2_vlan_allocations_vlan_id, ml2_vlan_allocations.allocated AS ml2_vlan_allocations_allocated \nFROM ml2_vlan_allocations' ()
2015-06-25 21:28:33.570 9129 TRACE neutron.service
2015-06-25 21:28:33.586 9129 CRITICAL neutron [-] OperationalError: (OperationalError) no such table: ml2_vlan_allocations u'SELECT ml2_vlan_allocations.physical_network AS ml2_vlan_allocations_physical_network, ml2_vlan_allocations.vlan_id AS ml2_vlan_allocations_vlan_id, ml2_vlan_allocations.allocated AS ml2_vlan_allocations_allocated \nFROM ml2_vlan_allocations' ()
lan_allocations' ()
2015-06-25 21:28:33.586 9129 TRACE neutron

root@alice:/var/log/neutron# service neutron-server status
neutron-server stop/waiting

This can be fixed if Neutron database migrations are applied manually like this:

sudo neutron-db-manage --config-file=/etc/neutron/neutron.conf upgrade head

Once the migration is done neutron-server logs show a new error "ImportError:
No module named neutron_lbaas.services.loadbalancer.plugin".

root@alice:/var/log/neutron# tail -n100 neutron-server.log

2015-06-25 21:33:31.154 7864 TRACE neutron.common.config raise err
2015-06-25 21:33:31.154 7864 TRACE neutron.common.config ImportError: No module named neutron_lbaas.services.loadbalancer.plugin
2015-06-25 21:33:31.154 7864 TRACE neutron.common.config
2015-06-25 21:33:31.158 7864 ERROR neutron.service [-] Unrecoverable error: please check log for details.
2015-06-25 21:33:31.158 7864 TRACE neutron.service Traceback (most recent call last):
2015-06-25 21:33:31.158 7864 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/neutron/service.py", line 103, in serve_wsgi
2015-06-25 21:33:31.158 7864 TRACE neutron.service service.start()
2015-06-25 21:33:31.158 7864 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/neutron/service.py", line 74, in start
2015-06-25 21:33:31.158 7864 TRACE neutron.service self.wsgi_app = _run_wsgi(self.app_name)
2015-06-25 21:33:31.158 7864 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/neutron/service.py", line 169, in _run_wsgi
2015-06-25 21:33:31.158 7864 TRACE neutron.service app = config.load_paste_app(app_name)
2015-06-25 21:33:31.158 7864 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/neutron/common/config.py", line 234, in load_paste_app
2015-06-25 21:33:31.158 7864 TRACE neutron.service raise RuntimeError(msg)
2015-06-25 21:33:31.158 7864 TRACE neutron.service RuntimeError: Unable to load neutron from configuration file /etc/neutron/api-paste.ini.

root@alice:/var/log/neutron# sudo apt-get install python-neutron-lbaas -y

Once you install this the error message is replaced with complains about the missing vpnaas module "ImportError: No module named neutron_vpnaas.services.vpn.service_drivers.ipsec".

root@alice:/var/log/neutron# tail -n100 neutron-server.log

2015-06-25 21:37:40.796 14496 TRACE neutron.common.config raise err
2015-06-25 21:37:40.796 14496 TRACE neutron.common.config ImportError: No module named neutron_vpnaas.services.vpn.service_drivers.ipsec
2015-06-25 21:37:40.796 14496 TRACE neutron.common.config
2015-06-25 21:37:40.808 14496 ERROR neutron.service [-] Unrecoverable error: please check log for details.
2015-06-25 21:37:40.808 14496 TRACE neutron.service Traceback (most recent call last):
2015-06-25 21:37:40.808 14496 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/neutron/service.py", line 103, in serve_wsgi
2015-06-25 21:37:40.808 14496 TRACE neutron.service service.start()
2015-06-25 21:37:40.808 14496 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/neutron/service.py", line 74, in start
2015-06-25 21:37:40.808 14496 TRACE neutron.service self.wsgi_app = _run_wsgi(self.app_name)
2015-06-25 21:37:40.808 14496 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/neutron/service.py", line 169, in _run_wsgi
2015-06-25 21:37:40.808 14496 TRACE neutron.service app = config.load_paste_app(app_name)
2015-06-25 21:37:40.808 14496 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/neutron/common/config.py", line 234, in load_paste_app
2015-06-25 21:37:40.808 14496 TRACE neutron.service raise RuntimeError(msg)
2015-06-25 21:37:40.808 14496 TRACE neutron.service RuntimeError: Unable to load neutron from configuration file /etc/neutron/api-paste.ini.

Obviously this can be fixed by install the vpnaas module:

root@alice:/var/log/neutron# sudo apt-get install python-neutron-vpnaas -y

IMHO nova-cloud-controller charm should take care of all these manual steps.

Here are the deatils about my setup:

alice: controller node
bob: compute node
charlie: quantum-gateway node
deploy: juju node

syed@deploy:~$ juju status
environment: manual
machines:
  "0":
    agent-state: started
    agent-version: 1.24.0
    dns-name: deploy.example.com
    instance-id: 'manual:'
    series: trusty
    hardware: arch=amd64 cpu-cores=2 mem=3953M
    state-server-member-status: has-vote
  "1":
    agent-state: started
    agent-version: 1.24.0
    dns-name: alice.example.com
    instance-id: manual:alice.example.com
    series: trusty
    hardware: arch=amd64 cpu-cores=2 mem=3953M
  "2":
    agent-state: started
    agent-version: 1.24.0
    dns-name: bob.example.com
    instance-id: manual:bob.example.com
    series: trusty
    hardware: arch=amd64 cpu-cores=2 mem=3953M
  "3":
    agent-state: started
    agent-version: 1.24.0
    dns-name: charlie.example.com
    instance-id: manual:charlie.example.com
    series: trusty
    hardware: arch=amd64 cpu-cores=2 mem=3953M
services:
  cinder:
    charm: cs:trusty/cinder-24
    exposed: false
    service-status:
      current: unknown
      since: 25 Jun 2015 21:24:58Z
    relations:
      amqp:
      - rabbitmq-server
      cinder-volume-service:
      - nova-cloud-controller
      cluster:
      - cinder
      identity-service:
      - keystone
      shared-db:
      - mysql
    units:
      cinder/0:
        workload-status:
          current: unknown
          since: 25 Jun 2015 21:24:58Z
        agent-status:
          current: idle
          since: 25 Jun 2015 21:34:24Z
          version: 1.24.0
        agent-state: started
        agent-version: 1.24.0
        machine: "1"
        public-address: alice.example.com
  glance:
    charm: cs:trusty/glance-22
    exposed: false
    service-status:
      current: unknown
      since: 25 Jun 2015 21:28:52Z
    relations:
      cluster:
      - glance
      identity-service:
      - keystone
      image-service:
      - nova-cloud-controller
      - nova-compute
      shared-db:
      - mysql
    units:
      glance/0:
        workload-status:
          current: unknown
          since: 25 Jun 2015 21:28:52Z
        agent-status:
          current: idle
          since: 25 Jun 2015 21:33:51Z
          version: 1.24.0
        agent-state: started
        agent-version: 1.24.0
        machine: "1"
        open-ports:
        - 9292/tcp
        public-address: alice.example.com
  heat:
    charm: cs:trusty/heat-7
    exposed: false
    service-status:
      current: unknown
      since: 25 Jun 2015 21:25:50Z
    relations:
      amqp:
      - rabbitmq-server
      identity-service:
      - keystone
      shared-db:
      - mysql
    units:
      heat/0:
        workload-status:
          current: unknown
          since: 25 Jun 2015 21:25:50Z
        agent-status:
          current: idle
          since: 25 Jun 2015 21:34:14Z
          version: 1.24.0
        agent-state: started
        agent-version: 1.24.0
        machine: "1"
        open-ports:
        - 8000/tcp
        - 8004/tcp
        public-address: alice.example.com
  keystone:
    charm: cs:trusty/keystone-26
    exposed: false
    service-status:
      current: unknown
      since: 25 Jun 2015 21:24:25Z
    relations:
      cluster:
      - keystone
      identity-service:
      - cinder
      - glance
      - heat
      - nova-cloud-controller
      - openstack-dashboard
      shared-db:
      - mysql
    units:
      keystone/0:
        workload-status:
          current: unknown
          since: 25 Jun 2015 21:24:25Z
        agent-status:
          current: idle
          since: 25 Jun 2015 21:52:01Z
          version: 1.24.0
        agent-state: started
        agent-version: 1.24.0
        machine: "1"
        public-address: alice.example.com
  mysql:
    charm: cs:trusty/mysql-25
    exposed: false
    service-status:
      current: unknown
      since: 25 Jun 2015 21:22:17Z
    relations:
      cluster:
      - mysql
      shared-db:
      - cinder
      - glance
      - heat
      - keystone
      - nova-cloud-controller
      - nova-compute
      - quantum-gateway
    units:
      mysql/0:
        workload-status:
          current: unknown
          since: 25 Jun 2015 21:22:17Z
        agent-status:
          current: idle
          since: 25 Jun 2015 21:30:57Z
          version: 1.24.0
        agent-state: started
        agent-version: 1.24.0
        machine: "2"
        public-address: bob.example.com
  nova-cloud-controller:
    charm: cs:trusty/nova-cloud-controller-58
    exposed: false
    service-status:
      current: unknown
      since: 25 Jun 2015 21:30:14Z
    relations:
      amqp:
      - rabbitmq-server
      cinder-volume-service:
      - cinder
      cloud-compute:
      - nova-compute
      cluster:
      - nova-cloud-controller
      identity-service:
      - keystone
      image-service:
      - glance
      quantum-network-service:
      - quantum-gateway
      shared-db:
      - mysql
    units:
      nova-cloud-controller/0:
        workload-status:
          current: unknown
          since: 25 Jun 2015 21:30:14Z
        agent-status:
          current: idle
          since: 25 Jun 2015 21:34:04Z
          version: 1.24.0
        agent-state: started
        agent-version: 1.24.0
        machine: "1"
        open-ports:
        - 3333/tcp
        - 8773/tcp
        - 8774/tcp
        - 9696/tcp
        public-address: alice.example.com
  nova-compute:
    charm: cs:trusty/nova-compute-24
    exposed: false
    service-status:
      current: unknown
      since: 25 Jun 2015 21:24:35Z
    relations:
      amqp:
      - rabbitmq-server
      cloud-compute:
      - nova-cloud-controller
      compute-peer:
      - nova-compute
      image-service:
      - glance
      shared-db:
      - mysql
    units:
      nova-compute/0:
        workload-status:
          current: unknown
          since: 25 Jun 2015 21:24:35Z
        agent-status:
          current: idle
          since: 25 Jun 2015 21:33:39Z
          version: 1.24.0
        agent-state: started
        agent-version: 1.24.0
        machine: "2"
        public-address: bob.example.com
  openstack-dashboard:
    charm: cs:trusty/openstack-dashboard-14
    exposed: false
    service-status:
      current: unknown
      since: 25 Jun 2015 21:51:33Z
    relations:
      cluster:
      - openstack-dashboard
      identity-service:
      - keystone
    units:
      openstack-dashboard/0:
        workload-status:
          current: unknown
          since: 25 Jun 2015 21:51:33Z
        agent-status:
          current: idle
          since: 25 Jun 2015 21:52:12Z
          version: 1.24.0
        agent-state: started
        agent-version: 1.24.0
        machine: "1"
        open-ports:
        - 80/tcp
        - 443/tcp
        public-address: alice.example.com
  quantum-gateway:
    charm: cs:trusty/quantum-gateway-16
    exposed: false
    service-status:
      current: unknown
      since: 25 Jun 2015 21:24:22Z
    relations:
      amqp:
      - rabbitmq-server
      cluster:
      - quantum-gateway
      quantum-network-service:
      - nova-cloud-controller
      shared-db:
      - mysql
    units:
      quantum-gateway/0:
        workload-status:
          current: unknown
          since: 25 Jun 2015 21:24:22Z
        agent-status:
          current: idle
          since: 25 Jun 2015 21:33:29Z
          version: 1.24.0
        agent-state: started
        agent-version: 1.24.0
        machine: "3"
        public-address: charlie.example.com
  rabbitmq-server:
    charm: cs:trusty/rabbitmq-server-32
    exposed: false
    service-status:
      current: unknown
      since: 25 Jun 2015 21:22:32Z
    relations:
      amqp:
      - cinder
      - heat
      - nova-cloud-controller
      - nova-compute
      - quantum-gateway
      cluster:
      - rabbitmq-server
    units:
      rabbitmq-server/0:
        workload-status:
          current: unknown
          since: 25 Jun 2015 21:22:32Z
        agent-status:
          current: idle
          since: 25 Jun 2015 21:33:01Z
          version: 1.24.0
        agent-state: started
        agent-version: 1.24.0
        machine: "1"
        open-ports:
        - 5672/tcp
        public-address: alice.example.com

[Edward Hope-Morley (hopem)]

Syed, support for neutron-server in the nova-cloud-controller charm was deprecated as of the 15.04 (see [1]) and moved to the neutron-api charm (lp:charms/trusty/neutron-api). Can you please confirm whether or not this problem exists when using the neutron-api charm?

[1] https://wiki.ubuntu.com/ServerTeam/OpenStackCharms/ReleaseNotes1504#nova-cloud-controller_and_neutron-api_charms

[Launchpad Janitor (janitor)]

[Expired for nova-cloud-controller (Juju Charms Collection) because there has been no activity for 60 days.]