Race between neutron-vpn-agent and neutron-ovs-cleanup

[Gema Gomez]

We had an issue on our stack today, neutron-vpn-agent failed to start due to neutron-ovs-cleanup not finishing cleanly:

Mar 21 08:56:24 kermath kernel: [ 86.711007] init: wait-for-state (neutron-vpn-agentneutron-ovs-cleanup) main process (1886) terminated with status 100
Mar 21 08:56:24 kermath kernel: [ 86.715325] init: neutron-vpn-agent pre-start process (1751) terminated with status 1

This left the stack without connectivity and no tenants could ssh into their bastions.

I would expect neutron-ovs-cleanup and neutron-vpn-agent to be able to recover from this on their own.

[James Page (james-page)]

Hi Gema

Please can you provide log files for both of the agents from the gateway unit.

Thanks!

[Gema Gomez (gema)]

cat vpn_agent.log
2016-03-21 11:27:39.015 41648 INFO oslo.messaging._drivers.impl_rabbit [req-cb34296e-1674-4ec6-83a9-ac57085cd2a8 - - - - -] Connecting to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:39.027 41648 INFO oslo.messaging._drivers.impl_rabbit [req-cb34296e-1674-4ec6-83a9-ac57085cd2a8 - - - - -] Connected to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:39.033 41648 INFO oslo.messaging._drivers.impl_rabbit [req-cb34296e-1674-4ec6-83a9-ac57085cd2a8 - - - - -] Connecting to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:39.045 41648 INFO oslo.messaging._drivers.impl_rabbit [req-cb34296e-1674-4ec6-83a9-ac57085cd2a8 - - - - -] Connected to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:39.267 41648 INFO oslo.messaging._drivers.impl_rabbit [req-9740db7e-1876-4fe4-8eaf-74ad9281e10c - - - - -] Connecting to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:39.292 41648 WARNING oslo_log.versionutils [-] Deprecated: WritableLogger() is deprecated as of Liberty and may be removed in Mitaka. It will not be superseded.
2016-03-21 11:27:39.294 41648 INFO oslo.messaging._drivers.impl_rabbit [req-9740db7e-1876-4fe4-8eaf-74ad9281e10c - - - - -] Connected to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:39.321 41648 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:39.334 41648 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:44.996 41648 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:45.008 41648 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:45.019 41648 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:45.028 41648 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:45.038 41648 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:45.048 41648 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:45.061 41648 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:45.076 41648 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:45.079 41648 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:45.080 41648 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:45.082 41648 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:45.084 41648 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:45.085 41648 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.230.19.137:5672
2016-03-21 11:27:45.087 41648 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.230.19.137:5672

cat ovs-cleanu...

[Gema Gomez (gema)]

Forgot to mention that we are running Liberty, the version of the charm in use is cs:trusty/neutron-gateway-9

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[James Page (james-page)]

Unfortunately I can't see from the provided log data why the neutron-ovs-cleanup task failed; without that information this is going to be tricky to debug - this is the last log data:

2016-03-21 08:56:42.852 1896 INFO neutron.cmd.ovs_cleanup [-] OVS cleanup completed successfully

The vpn agent is correctly waiting for the task to complete, so we need the detail of what went wrong in the cleanup task.

[James Page (james-page)]

Marking bug as Invalid as no response to #5 in > 12 months.