VPNaaS Missing Required Entries in Config Files

[Shannon McFarland]

In H.2 some of the required entries in various configuration files to enable VPNaaS are not being made.

The entries that are missing include:
/etc/neutron/rootwrap.d/vpnaas.filters
[Filters]
ip: IpFilter, ip, root
ip_exec: IpNetnsExecFilter, ip, root
openswan: CommandFilter, ipsec, root

/etc/neutron/vpn_agent.ini
[DEFAULT]
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

/etc/neutron/neutron.conf
[service_providers]
service_provider = VPN:Vpn:neutron.services.vpn.service_drivers.ipsec.IPsecVPNDriver:default
service_provider = FIREWALL:Iptables:neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver:default

I have verified that after puppet runs that the entries above are not in the files. Perhaps there some of these are not required but during a manual configuration of VPNaaS in H.1, adding these entries to the correctly entered entries that do work in H.2, VPNaaS works well and is stable.

[Shannon McFarland (shmcfarl)]

I got all of the debug/verbose stuff I could turned on and started walking
through the VPN config one item at a time and found this:

When you create and Ipsec connection the log yells about the root wrap:
2014-04-03 10:18:27.349 54507 TRACE
neutron.services.vpn.device_drivers.ipsec Stderr:
'/usr/bin/neutron-rootwrap: Unauthorized command: ip netns exec
qrouter-a0ffb720-7858-4da8-89bd-0aab2ca2cef6 ipsec pluto --ctlbase
/var/lib/neutron/ipsec/a0ffb720-7858-4da8-89bd-0aab2ca2cef6/var/run/pluto
--ipsecdir /var/lib/neutron/ipsec/a0ffb720-7858-4da8-89bd-0aab2ca2cef6/etc
--use-netkey --uniqueids --nat_traversal --secretsfile
/var/lib/neutron/ipsec/a0ffb720-7858-4da8-89bd-0aab2ca2cef6/etc/ipsec.secre
ts --virtual_private %v4:10.10.10.0/24,%v4:10.10.20.0/24 (no filter
matched)\n

Our H.2 does not create an entry in
/etc/neutron/rootwrap.d/vpnaas.filters. The working method I had before
had me add this:
[Filters]
ip: IpFilter, ip, root
ip_exec: IpNetnsExecFilter, ip, root
openswan: CommandFilter, ipsec, root

The log then changes to a new error I have not seen before which is about
the format of the vpnaas.filters file:
2014-04-03 10:20:48.273 60068 TRACE neutron Stderr: 'Traceback (most
recent call last):\n File "/usr/bin/neutron-rootwrap", line 10, in
<module>\n sys.exit(main())\n File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rootwrap/cmd.py"
, line 109, in main\n filters =
wrapper.load_filters(config.filters_path)\n File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rootwrap/wrapper
.py", line 114, in load_filters\n
filterconfig.read(os.path.join(filterdir, filterfile))\n File
"/usr/lib/python2.7/ConfigParser.py", line 305, in read\n
self._read(fp, filename)\n File "/usr/lib/python2.7/ConfigParser.py",
line 512, in _read\n raise MissingSectionHeaderError(fpname, lineno,
line)\nConfigParser.MissingSectionHeaderError: File contains no section
headers.\nfile: /etc/neutron/rootwrap.d/vpnaas.filters, line:
1\n\'lters]\\n\'\n'
2014-04-03 10:20:48.273 60068 TRACE neutron

[Pradeep Kilambi (pkilambi)]

Fix for interface_driver: https://review.openstack.org/#/c/85149/

The service_providers in neutron.conf are already set correctly by COI.

Still investigating rootwrap issue.

[Mark T. Voelker (mvoelker)]

So actually, the service_providers line gets set up correctly in Icehouse, but not in Havana. And that, I think is sort of a Neutron bug. Take a look here at the sample for Icehouse here:

https://github.com/CiscoSystems/neutron/blob/icehouse/etc/neutron.conf#L453

and for Havana:

https://github.com/CiscoSystems/neutron/blob/havana/etc/neutron.conf#L360

Note that in the latter, there's nothing for VPNaaS. The packages for Ubuntu use these sample files for what they drop into /etc/neutron when the .deb is installed, so they end up missing some stuff for VPNaaS.

IMHO this is a bit of a bug in Neutron Havana itself. However upstream may not accept this in stable branch. We could fix it for COI with a change to the sample file (a minor divergence from upstream). Or we could write a puppet provider to manage it (we'll likely want one eventually anyway). Or we could do a patch in the packaging code.

[Shannon McFarland (shmcfarl)]

Yeah, the service_provider entry is certainly not being made in the havana release. I am trying to figure out what neutron is after for the /etc/neutron/rootwrap.d/vpnaas.filters file. I have added what I have used in H.1 but perhaps something along the line has changed and more needs to be in that file as it keeps complaining about:

2014-04-03 10:20:48.273 60068 TRACE neutron Stderr: 'Traceback (most recent call last):\n File "/usr/bin/neutron-rootwrap", line 10, in <module>\n sys.exit(main())\n File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rootwrap/cmd.py", line 109, in main\n filters = wrapper.load_filters(config.filters_path)\n File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rootwrap/wrapper.py", line 114, in load_filters\n filterconfig.read(os.path.join(filterdir, filterfile))\n File "/usr/lib/python2.7/ConfigParser.py", line 305, in read\n self._read(fp, filename)\n File "/usr/lib/python2.7/ConfigParser.py", line 512, in _read\n raise MissingSectionHeaderError(fpname, lineno, line)\nConfigParser.MissingSectionHeaderError: File contains no section headers.\nfile: /etc/neutron/rootwrap.d/vpnaas.filters, line: 1\n\'lters]\\n\'\n'
2014-04-03 10:20:48.273 60068 TRACE neutron

[Shannon McFarland (shmcfarl)]

Major progress. I have a working VPN again. So a few things.

1) You absolutely have to have the rootwrap section in there:
/etc/neutron/rootwrap.d/vpnaas.filters
[Filters]
ip: IpFilter, ip, root
ip_exec: IpNetnsExecFilter, ip, root
openswan: CommandFilter, ipsec, root

2) You absolutely need have the interface driver in the vpn_agent.ini file. As I read the dev documentation (and I think you pointed this out also Mark) that this file 'should' inherit the l3_agent.ini config which has the interface driver defined but the log yells about it missing and when I add the driver line to the file the log completely cleans up and I have a working config again (with the below additions as well):
/etc/neutron/vpn_agent.ini
[DEFAULT]
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

3) As we have already discussed, we need the service_plugins (COI does this) AND the service_providers (COI does not add this in Havana):
/etc/neutron/neutron.conf
service_plugins =neutron.services.loadbalancer.plugin.LoadBalancerPlugin,neutron.services.firewall.fwaas_plugin.FirewallPlugin,neutron.services.vpn.plugin.VPNDriverPlugin

service_provider=LOADBALANCER:Haproxy:neutron.services.loadbalancer.drivers.haproxy.plugin_driver.HaproxyOnHostPluginDriver:default
service_provider = VPN:Vpn:neutron.services.vpn.service_drivers.ipsec.IPsecVPNDriver:default

With this setup I can create/destroy my VPNs and instances multiple times and have a clean test

Of course I can't reboot any of my nodes due to the https://bugs.launchpad.net/bugs/1292107

[Mark T. Voelker (mvoelker)]

> 2) You absolutely need have the interface driver in the vpn_agent.ini file.
> As I read the dev documentation (and I think you pointed this out also Mark)
> that this file 'should' inherit the l3_agent.ini config which has the interface
> driver defined but the log yells about it missing and when I add

This appears to be another area where the Ubuntu packaging has changed. In Havana I see the vpn agent getting started *without* "--config-file=/etc/neutron/l3_agent.ini", but in Icehouse it gets started *with* that argument. In the latter case it does appear to properly inherit the interface driver even if it's not specified in vpn_agent.ini. That said though:

1.) Pradeep's patch (which has now been merged for Icehouse...Pradeep, I think we may want this backported to stable/havana) means we don't have to care if the init script does the right thing or not.

2.) We want Pradeep's patch anyway since the are situations where the L3 agent isn't run (think provider network/HA scenarios).

[Ken Schroeder (kschroed)]

We have sen multiple issues with inheritance from the l3_agent.init. In addition o the interface_driver we have had a problem with the external_network_bridge since we use something other than default br-ex.

[DEFAULT]
# VPN-Agent configuration file
# Note vpn-agent inherits l3-agent, so you can use configs on l3-agent also
external_network_bridge=br-router
interface_driver=neutron.agent.linux.interface.OVSInterfaceDriver

[vpnagent]
vpn_device_driver=neutron.services.vpn.device_drivers.ipsec.OpenSwanDriver

[ipsec]
ipsec_status_check_interval=60

Do you guys have multiple service_provider lines configurations working then?

[Shannon McFarland (shmcfarl)]

Ken, I have two service provider lines, one for LBaaS/HAProxy and one for
VPNaaS.

On Fri, Apr 4, 2014 at 10:30 AM, Ken Schroeder <email address hidden> wrote:

> We have sen multiple issues with inheritance from the l3_agent.init. In
> addition o the interface_driver we have had a problem with the
> external_network_bridge since we use something other than default br-ex.
>
> [DEFAULT]
> # VPN-Agent configuration file
> # Note vpn-agent inherits l3-agent, so you can use configs on l3-agent also
> external_network_bridge=br-router
> interface_driver=neutron.agent.linux.interface.OVSInterfaceDriver
>
> [vpnagent]
> vpn_device_driver=neutron.services.vpn.device_drivers.ipsec.OpenSwanDriver
>
> [ipsec]
> ipsec_status_check_interval=60
>
>
> Do you guys have multiple service_provider lines configurations working
> then?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1301418
>
> Title:
> VPNaaS Missing Required Entries in Config Files
>
> Status in Openstack @ Cisco:
> In Progress
>
> Bug description:
> In H.2 some of the required entries in various configuration files to
> enable VPNaaS are not being made.
>
> The entries that are missing include:
> /etc/neutron/rootwrap.d/vpnaas.filters
> [Filters]
> ip: IpFilter, ip, root
> ip_exec: IpNetnsExecFilter, ip, root
> openswan: CommandFilter, ipsec, root
>
>
> /etc/neutron/vpn_agent.ini
> [DEFAULT]
> interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
>
> /etc/neutron/neutron.conf
> [service_providers]
> service_provider =
> VPN:Vpn:neutron.services.vpn.service_drivers.ipsec.IPsecVPNDriver:default
> service_provider =
> FIREWALL:Iptables:neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver:default
>
> I have verified that after puppet runs that the entries above are not
> in the files. Perhaps there some of these are not required but during
> a manual configuration of VPNaaS in H.1, adding these entries to the
> correctly entered entries that do work in H.2, VPNaaS works well and
> is stable.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/openstack-cisco/+bug/1301418/+subscriptions
>

[Pradeep Kilambi (pkilambi)]

Fix for external_network_bridge: https://review.openstack.org/85479

[Mark T. Voelker (mvoelker)]

https://github.com/CiscoSystems/neutron/pull/4 will fix the service_providers line in Havana.

[Mark T. Voelker (mvoelker)]

Looks like Ubuntu fixed the rootwrap stuff for Icehouse yesterday...we'll pull the updated package spec in a bit and add it to the havana branch too.

https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1303876

neutron (1:2014.1~rc1-0ubuntu2) UNRELEASED; urgency=medium

  * d/neutron-plugin-oneconvergence-agent.upstart: Rename upstart
    configuration to match package name (LP: #1301957).
  * d/neutron-vpn-agent.install: Install missing vpnaas.filters for
    rootwrap (LP: #1303876).

 -- James Page <email address hidden> Mon, 07 Apr 2014 16:16:52 +0100

[Mark T. Voelker (mvoelker)]

Here's the package spec backport for Havana. Please note: our build system is offline for a lab move temporarily, so it'll likely be later in the week before packages show up in the -proposed pocket for this.

http://bazaar.launchpad.net/~cisco-openstack/neutron/havana/revision/169

[Shannon McFarland (shmcfarl)]

Disregard my previous post on the base.pp error. That was an issue with my branch checkout.

I can validate that VPN is now working. I have deployed two AIOs using this patch, setup VPN and can successfully connect multiple instances together over the VPN.

Also, unlike before, the VPN will go 'active' immediately when configured vs. waiting for instances to go active.

Good work team!