Neutron service is reported down when Contrail is installed

[guillaume thouvenin]

Neutron service is reported down when Contrail is installed. The problem is that Contrail install neutron agents but they are not used. So neutron agent are not updated (ie field 'heartbeat_timestamp' is not updated in the DB). As we are querying the DB and check that this field is updated we detect that the neutron agents are down.

[guillaume thouvenin (guillaume-thouvenin)]

The command "neutron agent" is working and it returns no agent because contrail plugin returns an empty table. See
https://github.com/Juniper/contrail-neutron-plugin/blob/bdec44dac694834195f70437a7ee21508a2be96f/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin_base.py#L177

So information in the DB is not used.
To fix the problem on a deployed environment it is safe to delete the content of agents table:

# mysql
> use neutron;
> delete from agents;

[guillaume thouvenin (guillaume-thouvenin)]

The other solution is to modified the definition of the neutron cluster in the GSE template to only rely on neutron API if contrail plugin is deployed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/280594

[Patrick Petit (patrick-michel-petit)]

Or file a bug against Contrail to not install the agents if they are not used / started....

[Patrick Petit (patrick-michel-petit)]

I would not recommend to delete the agent entries in the DB.

[Simon Pasquier (simon-pasquier)]

Deleting the rows from the agents table is a quick and simple workaround if you've already deployed LMA + Contrail.

IMO the complete fix is two-fold:
- query the Neutron API instead of MySQL to get the status of the agents. This is true for the Nova/Cinder workers too.
- Tune the AFD/GSE configuration when Contrail is deployed.

[Andrey Grebennikov (agrebennikov)]

Simon, this workaround doesn't work.
I deleted the records from the DB:

mysql> delete from agents;
Query OK, 13 rows affected (0.02 sec)

But Grafana still shows neutron agents as Down (same number as before).
PS I restarted influxdb and grafana server just in case.

[Simon Pasquier (simon-pasquier)]

@Andrey, I think we forget to mention a required step in the workaround: you need to restart the lma_collector service on all the controllers too. Can you tell if it works better then?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/280594
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=7d86ec48e67c034bca5a92bc2ed0b46fb3975a6d
Submitter: Jenkins
Branch: master

commit 7d86ec48e67c034bca5a92bc2ed0b46fb3975a6d
Author: Guillaume Thouvenin <email address hidden>
Date: Tue Feb 16 11:29:57 2016 +0100

    Modify the definition of Neutron cluster when Contrail is installed

    This change removes the Neutron agents AFD when Contrail plugin is used
    to avoid reporting a DOWN status for Neutron. It also removes the
    collect of the metrics of the Neutron agents.

    Change-Id: I02ecb67489d244aca85bc4b1e3d4a5cd79df1b5b
    Closes-Bug: #1546017

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/0.8
Review: https://review.openstack.org/283602

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/0.9
Review: https://review.openstack.org/283603

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/283603
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=a7278433b317b06a9040e5cd32091db5f8114839
Submitter: Jenkins
Branch: stable/0.9

commit a7278433b317b06a9040e5cd32091db5f8114839
Author: Guillaume Thouvenin <email address hidden>
Date: Tue Feb 16 11:29:57 2016 +0100

    Modify the definition of Neutron cluster when Contrail is installed

    This change removes the Neutron agents AFD when Contrail plugin is used
    to avoid reporting a DOWN status for Neutron. It also removes the
    collect of the metrics of the Neutron agents.

    Change-Id: I02ecb67489d244aca85bc4b1e3d4a5cd79df1b5b
    Closes-Bug: #1546017
    (cherry picked from commit 7d86ec48e67c034bca5a92bc2ed0b46fb3975a6d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/283602
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=1179e36b300369801ca3c575e95cb3e054960689
Submitter: Jenkins
Branch: stable/0.8

commit 1179e36b300369801ca3c575e95cb3e054960689
Author: Guillaume Thouvenin <email address hidden>
Date: Tue Feb 16 11:29:57 2016 +0100

    Modify the definition of Neutron cluster when Contrail is installed

    This change removes the Neutron agents AFD when Contrail plugin is used
    to avoid reporting a DOWN status for Neutron. It also removes the
    collect of the metrics of the Neutron agents.

    Conflicts:
     deployment_scripts/puppet/manifests/controller.pp

    Change-Id: I02ecb67489d244aca85bc4b1e3d4a5cd79df1b5b
    Closes-Bug: #1546017
    (cherry picked from commit da5a56fcc4b0e208e5ded8a38f6193a3e31e1c68)